Data kinship processing method and device based on data warehouse

Abstract

The present application discloses a data lineage processing method and device based on a data warehouse. The method uses a preset regular expression to obtain the inclusion relationship between each work set and the inclusion relationship between the work set and the ETL task from the character information of the work set. The character information of the work set is parsed to obtain the dependency relationship between each ETL task. The ETL task is parsed to obtain the components contained in the ETL task. Each component is parsed to obtain the dependency relationship between each library table and the dependency relationship between each field contained in each component. Based on the dependency relationship between each library table contained in each component, the dependency relationship between each library table in the data warehouse is constructed. Compared with the prior art, the scheme shown in the present application does not require manual participation in the data lineage parsing process, which saves a lot of manpower costs, and the parsed data lineage relationship is more accurate and real-time.

Description

Technical Field

[0001] The present application relates to the technical field of data warehouses, and in particular to a method and device for processing data lineage relationships based on a data warehouse. Background Art

[0002] As the number of business product lines of enterprises continues to increase, the underlying data records are becoming more and more complex, and the construction of enterprise-level data warehouses has become more important. In the process of data warehouse construction, multiple people participating in development has become the norm, and the number of extraction-transformation-loading (ETL) tasks has gradually increased to hundreds. There are many difficulties in maintenance, especially the data lineage relationship between ETL tasks and data warehouse tables (referred to as library tables) is particularly complex.

[0003] At present, the data lineage relationship of data warehouse is generally maintained manually, that is, manually querying the lineage relationship between ETL tasks and database tables. Obviously, due to human factors, the query process of data lineage relationship of data warehouse will take a lot of time, its query efficiency is low, and it is more prone to errors. Summary of the invention

[0004] The present application provides a data lineage processing method and device based on a data warehouse, aiming to help users quickly and effectively learn about the data lineage contained in the data warehouse.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] A data kinship processing method based on a data warehouse, comprising:

[0007] Get the working sets and ETL tasks contained in the data warehouse;

[0008] Using a preset regular expression, the inclusion relationship between the work sets and the inclusion relationship between the work sets and the ETL task are obtained from the character information of the work sets;

[0009] Parsing the character information of the work set to obtain the dependency relationship between each of the ETL tasks;

[0010] Parsing the ETL task to obtain various components included in the ETL task;

[0011] Parsing each of the components to obtain each library table contained in each component, dependency relationships between the library tables, each field, and dependency relationships between the fields;

[0012] Based on the dependency relationship between the library tables contained in each component, construct the dependency relationship between the library tables in the data warehouse;

[0013] Based on the dependency relationship between the fields contained in each component, construct the dependency relationship between the fields in the data warehouse;

[0014] Based on each of the library tables included in the ETL task, generating an inclusion relationship between the ETL task and the library table;

[0015] Acquire the fields included in each of the library tables, and based on the fields included in each of the library tables, generate an inclusion relationship between the library tables and the fields;

[0016] A valid word set of each of the library tables is obtained, and based on the valid word set of each of the library tables, a semantic similarity relationship between each of the library tables is determined.

[0017] Optionally, obtaining the working set and ETL tasks contained in the data warehouse includes:

[0018] Obtain the working sets and ETL tasks contained in the data warehouse from the pre-built third information table;

[0019] Among them, the construction process of the third information table includes: obtaining each project file of the data warehouse; identifying the project files with the file suffix .kjb in each of the project files as the working set, and identifying the project files with the file suffix .ktr as the ETL task; and constructing the third information table based on each of the working sets and the ETL tasks.

[0020] Optionally, the using of a preset regular expression to obtain the inclusion relationship between the respective work sets and the inclusion relationship between the work set and the ETL task from the character information of the work set includes:

[0021] Acquire a character string matching a first regular expression from the character information of the working set;

[0022] From the strings that match the first regular expression, obtain the strings that match the second regular expression;

[0023] Obtain the character string that matches the third regular expression from the character information of the working set;

[0024] Acquire a string matching a fourth regular expression from the string matching the second regular expression, and parse the string matching the fourth regular expression to obtain other work sets included in the work set;

[0025] Acquire a string matching a fifth regular expression from the string matching the third regular expression, and parse the string matching the fifth regular expression to obtain the ETL task included in the work set;

[0026] Based on other work sets contained in each work set, generating an inclusion relationship between the work sets;

[0027] Based on the ETL tasks included in each of the work sets, an inclusion relationship between the work sets and the ETL tasks is generated.

[0028] Optionally, parsing the character information of the work set to obtain the dependency relationship between each of the ETL tasks includes:

[0029] Acquire a character string matching the first keyword from the character information of the work set;

[0030] From the character strings that match the first keyword, obtain the character strings that match the second keyword;

[0031] Parsing the string that matches the second keyword to obtain the name of the source task and the name of the target task that depends on the source task;

[0032] Acquire a character string matching a third keyword from the character information of the work set;

[0033] Parsing the character string that matches the third keyword to obtain each file path;

[0034] Acquire a file path including the name of the source task and a file path including the name of the target task from each of the file paths;

[0035] According to the file path containing the name of the source task, query and obtain the ETL task corresponding to the source task;

[0036] According to the file path containing the name of the target task, query and obtain the ETL task corresponding to the target task;

[0037] Based on the dependency relationship between the source task and the target task, the ETL task corresponding to the source task, and the ETL task corresponding to the target task, a dependency relationship between the ETL tasks is generated.

[0038] Optionally, the type of the component includes a stored procedure;

[0039] The parsing of each component to obtain each library table contained in each component, the dependency relationship between each library table, each field, and the dependency relationship between each field includes:

[0040] For each stored procedure included in the ETL task, obtaining the effective implementation logic of the stored procedure from a pre-constructed fourth information table, and marking the effective implementation logic as the logic to be parsed;

[0041] Among them, the construction process of the fourth information table includes: pre-acquiring each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the outline of the project logic; calling the preset SQL query statement to query from the outline to obtain the implementation logic of each stored procedure contained in the project logic; performing data cleaning on the implementation logic of each stored procedure to obtain the effective implementation logic of each stored procedure; performing data format conversion on the effective implementation logic of each stored procedure so that the data format of the effective implementation logic of each effective stored procedure is the same; based on the effective implementation logic of each stored procedure after the data format conversion, constructing the fourth information table;

[0042] The logic to be analyzed is analyzed to obtain various library tables, dependency relationships between the library tables, various fields, and dependency relationships between the fields contained in the storage process.

[0043] Optionally, parsing the logic to be parsed to obtain various library tables, dependencies between the library tables, various fields, and dependencies between the fields contained in the storage process includes:

[0044] Acquire each character string matching the fourth keyword from the logic to be parsed;

[0045] Parsing each character string that matches the fourth keyword to obtain each output table included in the logic to be parsed;

[0046] Acquire each character string matching the fifth keyword from the logic to be parsed;

[0047] Parsing each character string that matches the fifth keyword to obtain each library table included in the logic to be parsed;

[0048] Using the library tables whose table names are different from the table names of the output tables as the input tables included in the logic to be parsed;

[0049] Pair each output table included in the logic to be parsed with each input table included in the logic to be parsed, to obtain the dependency relationship between the library tables included in the storage process;

[0050] Acquire each character string matching the sixth keyword from the logic to be parsed;

[0051] Parsing each character string that matches the sixth keyword to obtain each output field included in the logic to be parsed;

[0052] Acquire each character string matching the seventh keyword from the logic to be parsed;

[0053] Parsing each character string that matches the seventh keyword to obtain each candidate field included in the to-be-parsed logic;

[0054] For each of the candidate fields, when it is determined that the candidate field is recorded in the pre-constructed first information table, the candidate field is used as an input field included in the logic to be parsed;

[0055] Among them, the construction process of the first information table includes: obtaining each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the data warehouse information of the project logic; calling the preset SQL query statement to query and obtain the metadata information table of the project logic from the data warehouse information; the metadata information table includes various meta information; each meta information includes a library table, the table name and description information of the library table, the fields contained in the library table, the field name and description information of the field; performing data cleaning on each meta information to obtain each valid meta information; performing data format conversion on each valid meta information so that the data format of each valid meta information is the same; based on each valid meta information after the data format conversion, constructing the first information table;

[0056] Using the field name of the input field as a search basis, searching the field name of each output field to obtain the input field that each output field depends on;

[0057] Based on the input fields that each of the output fields depends on, a dependency relationship between the fields is generated.

[0058] Optionally, the type of the component includes a table output component;

[0059] The parsing of each component to obtain each library table contained in each component, the dependency relationship between each library table, each field, and the dependency relationship between each field includes:

[0060] For each table output component included in the ETL task, obtaining each character string matching the eighth keyword from the table output component;

[0061] Parsing each character string matched with the eighth keyword to obtain each output table included in the table output component;

[0062] Acquire each character string matching the ninth keyword from the table output component;

[0063] Parsing each character string matched with the ninth keyword to obtain each input table included in the table output component;

[0064] Pairing each output table included in the table output component with each input table included in the table output component to obtain dependency relationships between various library tables;

[0065] Acquire each character string matching the sixth keyword from the table output component;

[0066] Parsing each character string matching the sixth keyword to obtain each output field included in the table output component;

[0067] Obtaining from the table output component each character string that matches the seventh keyword;

[0068] Parsing each character string matched with the seventh keyword to obtain each candidate field included in the table output component;

[0069] For each of the candidate fields, when it is determined that the candidate field is recorded in the pre-constructed first information table, the candidate field is used as an input field included in the table output component;

[0070] Using the field name of the input field as a search basis, searching the field name of each output field to obtain the input field that each output field depends on;

[0071] Based on the input fields that each of the output fields depends on, a dependency relationship between the fields is generated.

[0072] Optionally, the acquiring of valid word sets of each of the library tables, and determining semantic similarity relationships between each of the library tables based on the valid word sets of each of the library tables, includes:

[0073] For each of the library tables included in the ETL task, obtain the table name and description information of the library table from a pre-built first information table;

[0074] Segmenting the table name and description information of the library table respectively to obtain a candidate word set of the library table; the candidate word set includes multiple candidate words;

[0075] Acquire a synonym set corresponding to each of the candidate words from a pre-constructed second information table; the synonym set includes a plurality of synonyms;

[0076] The construction process of the second information table includes: obtaining each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the data warehouse information of the project logic; calling a preset SQL query statement to query and obtain the metadata information table of the project logic from the data warehouse information; the metadata information table includes various meta information; each meta information includes a library table, a field contained in the library table, a field name and description information of the field; performing data cleaning on each meta information to obtain each valid meta information; performing semantic recognition on each valid meta information to obtain each business noun, and configuring a synonym set corresponding to each business noun; based on each business noun and a synonym set corresponding to each business noun, constructing the second information table;

[0077] From each of the candidate words, select a candidate word whose word type is a noun and mark it as a valid word;

[0078] From the synonyms corresponding to each of the candidate words, a synonym of a verb type is selected and marked as a valid word;

[0079] Based on each of the valid words, construct a valid word set in the library table;

[0080] Input valid words in the first valid word set and valid words in the second valid word set into a preset word similarity model to obtain each triple pair output by the word similarity model; the first valid word set is a valid word set of the first library table; the second valid word set is a valid word set of the second library table; the first library table and the second library table are both library tables included in the ETL task; the triple pair includes an original word, a most similar word corresponding to the original word, and a similarity related to the original word;

[0081] Calculating the mean of each of the similarities to obtain the library table similarity between the first library table and the second library table;

[0082] When the library table similarity is greater than a preset threshold, it is determined that there is a semantically similar relationship between the first library table and the second library table.

[0083] Optionally, also include:

[0084] The inclusion relationship between the various working sets, the inclusion relationship between the working sets and the ETL tasks, the dependency relationship between the various ETL tasks, the dependency relationship between the various library tables, the dependency relationship between the various fields, the inclusion relationship between the ETL tasks and the library tables, the inclusion relationship between the library tables and the fields, and the semantic similarity relationship between the various library tables are saved in the preset library table as the data lineage relationship of the data warehouse;

[0085] The data blood relationship recorded in the preset library table is converted into a graph to obtain a relationship graph, and the relationship graph is displayed to the user through a preset interface.

[0086] A data kinship processing device based on a data warehouse, comprising:

[0087] A task acquisition unit is used to acquire the working set and ETL tasks contained in the data warehouse;

[0088] A regular analysis unit, used for obtaining the inclusion relationship between each of the work sets and the inclusion relationship between the work set and the ETL task from the character information of the work set by using a preset regular expression;

[0089] A work set parsing unit, used to parse the character information of the work set to obtain the dependency relationship between each of the ETL tasks;

[0090] A task parsing unit, used to parse the ETL task to obtain various components included in the ETL task;

[0091] A component parsing unit, used for parsing each of the components to obtain each library table contained in each component, the dependency relationship between each of the library tables, each field, and the dependency relationship between each of the fields;

[0092] A first construction unit, configured to construct a dependency relationship between the library tables in the data warehouse based on the dependency relationship between the library tables contained in each component;

[0093] A second construction unit, configured to construct a dependency relationship between the fields in the data warehouse based on the dependency relationship between the fields included in each component;

[0094] A first generating unit, configured to generate an inclusion relationship between the ETL task and the library table based on each of the library tables included in the ETL task;

[0095] A second generating unit, used for acquiring the fields included in each of the library tables, and generating an inclusion relationship between the library table and the fields based on the fields included in each of the library tables;

[0096] The determination unit is used to obtain the valid word sets of each of the library tables, and determine the semantic similarity relationship between each of the library tables based on the valid word sets of each of the library tables.

[0097] The technical solution provided by the present application obtains the working sets and ETL tasks contained in the data warehouse. Using a preset regular expression, the inclusion relationship between each working set and the inclusion relationship between the working set and the ETL task are obtained from the character information of the working set. The character information of the working set is parsed to obtain the dependency relationship between each ETL task. The ETL task is parsed to obtain the various components contained in the ETL task. Each component is parsed to obtain the various library tables contained in each component, the dependency relationship between each library table, each field, and the dependency relationship between each field. Based on the dependency relationship between each library table contained in each component, the dependency relationship between each library table in the data warehouse is constructed. Based on the dependency relationship between each field contained in each component, the dependency relationship between each field in the data warehouse is constructed. Based on each library table contained in the ETL task, the inclusion relationship between the ETL task and the library table is generated. The fields contained in each library table are obtained, and based on the fields contained in each library table, the inclusion relationship between the library table and the field is generated. Obtain the valid word set of each library table, and determine the semantic similarity relationship between each library table based on the valid word set of each library table. Using the scheme shown in this application, users can quickly and effectively learn the data lineage relationship contained in the data warehouse. Compared with the existing technology, there is no need for manual participation in the data lineage relationship analysis process, which saves a lot of manpower costs, and the data lineage relationship obtained by analysis is more accurate and real-time. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0099] Figure 1a A flowchart of a data kinship processing method based on a data warehouse provided in an embodiment of the present application;

[0100] Figure 1b A flowchart of a data kinship processing method based on a data warehouse provided in an embodiment of the present application;

[0101] Figure 1c A flowchart of a data kinship processing method based on a data warehouse provided in an embodiment of the present application;

[0102] Figure 1d A flowchart of a data kinship processing method based on a data warehouse provided in an embodiment of the present application;

[0103] Figure 2 A flowchart of another method for processing data lineage relationships based on a data warehouse provided in an embodiment of the present application;

[0104] Figure 3 A schematic diagram of the architecture of a data lineage relationship processing device based on a data warehouse provided in an embodiment of the present application. DETAILED DESCRIPTION

[0105] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0106] like Figure 1a , Figure 1b , Figure 1c and Figure 1d FIG. 1 is a flow chart of a method for processing data lineage relationships based on a data warehouse provided in an embodiment of the present application, comprising the following steps:

[0107] S101: Pre-acquire various project logics and project files of the data warehouse, as well as configuration files of each project logic.

[0108] Among them, the various project logics and project files of the data warehouse, as well as the configuration files of each project logic, can be directly pulled from the code repository of the data warehouse.

[0109] S102: For each project logic, parse the configuration file of the project logic to obtain data warehouse information and schema of the project logic.

[0110] S103: calling a preset SQL query statement to query and obtain a metadata information table of the project logic from the data warehouse information.

[0111] Among them, the metadata information table includes various meta information, including but not limited to: server IP, environment name, data warehouse name, library table, table name and description information of the library table, fields contained in the library table, field name and description information of the field, field type, module to which the field belongs, etc.

[0112] S104: Perform data cleaning on each piece of meta information to obtain each piece of valid meta information.

[0113] The specific implementation process of data cleaning is common knowledge familiar to those skilled in the art and will not be described in detail here.

[0114] S105: Convert the data format of each valid meta information so that the data format of each valid meta information is the same.

[0115] The specific implementation process of data format conversion is common knowledge familiar to those skilled in the art and will not be described in detail here.

[0116] S106: Construct a first information table based on each valid meta information after data format conversion.

[0117] S107: Perform semantic recognition on each valid meta-information to obtain each business noun, and configure a synonym set corresponding to each business noun.

[0118] The synonym set includes multiple synonyms, and the synonyms corresponding to the business terms can be obtained from the Internet, or can be set by technical personnel according to actual conditions.

[0119] It should be noted that the specific implementation process of semantic recognition is common knowledge familiar to those skilled in the art and will not be elaborated here.

[0120] S108: Constructing a second information table based on each business noun and a synonym set corresponding to each business noun.

[0121] The specific form of the second information table can be seen in Table 1.

[0122] Table 1

[0123] Business terms Synonym collection White Strip [Rongxin, E-Xintong, Reverse Factoring] Order Financing [Flow loan, E-sales, mortgage loan] Financing Amount [Amount of investment, amount of loan before interest deduction] Expiry date of the IOU [Promised payment date, payment date] Core Enterprises [Source of the credit limit, issuing party] Chain Enterprise [Financiers, suppliers]

[0124] It should be noted that the contents shown in Table 1 above are only for illustration.

[0125] S109: Among the project files, a project file with a file suffix of .kjb is identified as a working set, and a project file with a file suffix of .ktr is identified as an ETL task.

[0126] Among them, a project file with a file suffix of .kjb usually encapsulates multiple project files with a file suffix of .ktr, or encapsulates other project files with a file suffix of .kjb. A project file with a file suffix of .ktr is actually a Kettle code file, that is, an ETL task for developing data warehouse tables. The implementation logic of the ETL task is: the character information contained in the project file with a file suffix of .ktr.

[0127] Generally speaking, the data warehouse described in the embodiment of the present application is a data warehouse that uses the Kettle open source tool as a development tool. Therefore, each ETL task contained in the data warehouse uses various types of components (including table input components, table output components, stored procedures, and SQL components) provided by the Kettle open source tool. In the embodiment of the present application, the so-called component can be simply understood as pre-packaged logic.

[0128] It should be emphasized that the role of the Kettle code file (that is, the role of the ETL task) is to write development logic through the fields of multiple input tables and generate the fields in the corresponding output table. The so-called input tables and output tables are both data warehouse tables.

[0129] S110: Construct a third information table based on each work set and ETL task.

[0130] S111: calling a preset SQL query statement to query from the outline and obtain the implementation logic of each stored procedure included in the project logic.

[0131] Optionally, you can also call a preset SQL query statement to query the implementation logic contained in each view of the project logic from the outline.

[0132] S112: Perform data cleansing on the implementation logic of each stored procedure to obtain the effective implementation logic of each stored procedure.

[0133] The information involved in implementing the logic includes, but is not limited to: server IP, environment name, data warehouse name, task name, code, etc.

[0134] S113: Convert the data format of the effective implementation logic of each stored procedure so that the data format of the effective implementation logic of each effective stored procedure is the same.

[0135] S114: Constructing a fourth information table based on the effective implementation logic of each stored procedure after data format conversion.

[0136] S115: For each work set recorded in the third information table, using a preset regular expression, obtain the inclusion relationship between each work set and the inclusion relationship between the work set and the ETL task from the character information of the work set.

[0137] The specific implementation process of using the preset regular expression to obtain the inclusion relationship between each work set from the character information of the work set, and the inclusion relationship between the work set and the ETL task is as follows:

[0138] 1. Get the string that matches the first regular expression from the character information of the working set.

[0139] Specifically, the first regular expression can be set to: <entry> …< / entry> .

[0140] 2. From the strings that match the first regular expression, get the strings that match the second regular expression.

[0141] Specifically, the second regular expression can be set to: <type> JOB< / type> .

[0142] 3. Get the string that matches the third regular expression from the character information of the working set.

[0143] Specifically, the third regular expression can be set as: <type> TRANS< / type> .

[0144] 4. From the string that matches the second regular expression, obtain the string that matches the fourth regular expression, and parse the string that matches the fourth regular expression to obtain other working sets included in the working set.

[0145] Specifically, the fourth regular expression can be set as: <jobname> …< / jobname> .

[0146] 5. From the string that matches the third regular expression, obtain the string that matches the fifth regular expression, and parse the string that matches the fifth regular expression to obtain the ETL tasks included in the work set.

[0147] Specifically, the fifth regular expression can be set as: <transname> …< / transname> .

[0148] 6. Based on the other working sets contained in each working set, generate the inclusion relationship between each working set.

[0149] 7. Based on the ETL tasks included in each work set, generate the inclusion relationship between the work set and the ETL tasks.

[0150] S116: Parse the character information of the work set to obtain the dependency relationship between each ETL task.

[0151] The specific implementation process of parsing the character information of the working set is as follows:

[0152] 1. Obtain a character string that matches the first keyword from the character information of the working set.

[0153] Specifically, the first keyword can be set to: <hops>.

[0154] 2. From the character strings that match the first keyword, obtain the character strings that match the second keyword.

[0155] Specifically, the second keyword can be set to: <hop>.

[0156] 3. Parse the string that matches the second keyword to obtain the name of the source task and the name of the target task that depends on the source task.

[0157] 4. Obtain a character string matching the third keyword from the character information of the working set.

[0158] Specifically, the third keyword can be set to: <entry>。

[0159] 5. Parse each file path from the strings that match the third keyword.

[0160] 6. Obtain the file paths containing the names of the source tasks and the file paths containing the names of the target tasks from each file path.

[0161] 7. Query the ETL task corresponding to the source task according to the file path containing the name of the source task.

[0162] 8. Query the ETL task corresponding to the target task according to the file path containing the name of the target task.

[0163] 9. Generate the dependency relationships between each ETL task based on the dependency relationship between the source task and the target task, the ETL task corresponding to the source task, and the ETL task corresponding to the target task.

[0164] S117: For each ETL task recorded in the third information table, parse the ETL task to obtain the stored procedures, SQL components, table output components, and table input components included in the ETL task.

[0165] S118: Obtain the effective implementation logic of the stored procedure included in the ETL task from the fourth information table, and identify the effective implementation logic of the stored procedure included in the ETL task as the logic to be parsed.

[0166] S119: Parse the logic to be parsed to obtain each library table included in the stored procedure, the dependency relationships between each library table, each field, and the dependency relationships between each field.

[0167] Among them, the types of library tables include output tables and input tables, and the types of fields include output fields and input fields.

[0168] It should be noted that the specific implementation process of parsing the logic to be parsed is as follows:

[0169] 1. Obtain each string that matches the fourth keyword from the logic to be parsed.

[0170] Specifically, the fourth keyword can be set as: INTO|UPDATE + <Schema list> +. + .

[0171] 2. Parse each output table included in the logic to be parsed from each string that matches the fourth keyword.

[0172] Optionally, in order to ensure that each output table included in the logic to be parsed is unique, redundant output tables in each output table can also be deleted.

[0173] 3. Obtain each string that matches the fifth keyword from the logic to be parsed.

[0174] Specifically, the fifth keyword can be set as: <Schema list> +. + .

[0175] 4. Parse and obtain each library table included in the logic to be parsed from each string that matches the fifth keyword.

[0176] 5. Use the library tables whose table names are different from the table name of the output table in each library table as the input tables included in the logic to be parsed.

[0177] Optionally, to ensure that each input table included in the logic to be parsed is unique, redundant input tables in each input table can also be deleted.

[0178] 6. Pair each output table included in the logic to be parsed with each input table included in the logic to be parsed respectively to obtain the dependency relationships between the various library tables included in the stored procedure.

[0179] 7. Obtain each string that matches the sixth keyword from the logic to be parsed.

[0180] Specifically, the sixth keyword can be set as: INTO + <Schema list> +. + + <field name list>.

[0181] 8. Parse and obtain each output field included in the logic to be parsed from each string that matches the sixth keyword.

[0182] Specifically, in the string that matches the sixth keyword, there is one or more preset characters (specifically ",", and the string between two preset characters is the so-called output field.

[0183] 9. Obtain each string that matches the seventh keyword from the logic to be parsed.

[0184] Specifically, the seventh keyword can be set as: SELECT + <field name list> + FROM + <Schema list> +. + .

[0185] 10. Parse and obtain each candidate field included in the logic to be parsed from each string that matches the seventh keyword.

[0186] 11. For each candidate field, when the candidate field is recorded in the first information table, use the candidate field as the input field included in the logic to be parsed.

[0187] It should be noted that, when no candidate field is recorded in the first information table, the candidate field is identified as a virtual field and the virtual field is deleted.

[0188] 12. Use the field name of the input field as the search basis, search the field name of each output field, and obtain the input field that each output field depends on.

[0189] It should be noted that the so-called field name search includes but is not limited to: recursively searching the field name of the input field in the output field according to the order from the outside to the inside of the SQL structure, and finding the dependent field name contained in the innermost structure of the output field, which is the input field that the output field depends on.

[0190] 13. Based on the input fields that each output field depends on, generate the dependency relationship between the fields.

[0191] S120: Parse the SQL component to obtain various library tables and dependency relationships between the library tables, and various fields and dependency relationships between the fields contained in the SQL component.

[0192] Among them, the specific execution process and implementation principle of parsing the SQL component are consistent with the specific implementation process and implementation principle of parsing the parsing logic shown in S119, and will not be repeated here.

[0193] S121: Parse the table output component to obtain various library tables and dependency relationships between the library tables, and various fields and dependency relationships between the fields contained in the table output component.

[0194] The specific implementation process of parsing the table output component is as follows:

[0195] 1. Obtain each string matching the eighth keyword from the table output component.

[0196] Specifically, the eighth keyword can be set to: <tableoutput>。

[0197] 2. Parse each output table included in the table output component from each string that matches the eighth keyword.

[0198] 3. Obtain each string that matches the ninth keyword from the table output component.

[0199] Specifically, the ninth keyword can be set as: <Schema list> +.+ .

[0200] 4. Parse each input table included in the table output component from each string that matches the ninth keyword.

[0201] 5. Pair each output table included in the table output component with each input table included in the table output component respectively to obtain the dependency relationships between each library table.

[0202] 6. Obtain each string that matches the sixth keyword from the table output component.

[0203] 7. Parse each output field included in the table output component from each string that matches the sixth keyword.

[0204] 8. Obtain each string that matches the seventh keyword from the table output component.

[0205] 9. Parse each candidate field included in the table output component from each string that matches the seventh keyword.

[0206] 10. For each candidate field, when the candidate field is recorded in the first information table, use the candidate field as the input field included in the table output component.

[0207] 11. Use the field name of the input field as the search criterion to perform a field name search on each output field to obtain the input field on which each output field depends.

[0208] 12. Generate the dependency relationships between each field based on the input field on which each output field depends.

[0209] S122: Parse the table input component to obtain each library table, the dependency relationships between each library table, each field, and the dependency relationships between each field included in the table input component.

[0210] Among them, the specific execution process and implementation principle for parsing the table input component are the same as the specific implementation process and implementation principle for parsing the table output component shown in S121 above, and will not be elaborated here.

[0211] S123: Based on the dependencies between the libraries and tables contained in the logic to be parsed, the dependencies between the libraries and tables contained in the SQL component, the dependencies between the libraries and tables contained in the table output component, and the dependencies between the libraries and tables contained in the table input component, the dependencies between the libraries and tables in the data warehouse are constructed.

[0212] S124: Based on the dependencies between the fields contained in the logic to be parsed, the dependencies between the fields contained in the SQL component, the dependencies between the fields contained in the table output component, and the dependencies between the fields contained in the table input component, the dependencies between the fields in the data warehouse are constructed.

[0213] S125: Based on the library tables included in the logic to be parsed, the library tables included in the SQL component, the library tables included in the table output component, and the library tables included in the table input component, the library tables included in the ETL task are constructed.

[0214] S126: Based on each library table included in the ETL task, an inclusion relationship between the ETL task and the library table is generated.

[0215] S127: For each library table included in the ETL task, query the fields included in the library table from the first information table.

[0216] S128: Based on the fields included in the library table, generate an inclusion relationship between the library table and the fields.

[0217] S129: Obtain the table name and description information of the library table from the first information table.

[0218] S130: Segment the table name and description information of the library table respectively to obtain a candidate word set of the library table.

[0219] The specific implementation process of word segmentation is common knowledge familiar to those skilled in the art and will not be described in detail here. In the embodiment of the present application, the candidate word set includes a plurality of candidate words.

[0220] S131: Obtain a synonym set corresponding to each candidate word from the second information table.

[0221] The synonym set includes multiple synonyms.

[0222] S132: Select candidate words whose word type is noun from each candidate word, and mark them as valid words.

[0223] S133: From the synonyms corresponding to each candidate word, a synonym of the word type of verb is selected and marked as a valid word.

[0224] S134: Based on each valid word, construct a valid word set in the library table.

[0225] S135: Based on the valid word sets of each library table, determine the semantic similarity relationship between each library table and the similarity label of each library table.

[0226] Among them, based on the valid word set of each library table, the specific implementation process of determining the semantic similarity relationship between each library table and the similarity label of each library table is as follows:

[0227] 1. Input the valid words in the first valid word set and the valid words in the second valid word set into a preset word similarity model to obtain each triple pair output by the word similarity model.

[0228] It should be noted that the first valid word set is a valid word set of the first library table, and the second valid word set is a valid word set of the second library table. Both the first library table and the second library table are library tables included in the ETL task.

[0229] In addition, the word similarity model is used to calculate the similarity between each valid word and other valid words. The so-called word similarity model is common knowledge familiar to those skilled in the art, including but not limited to: Logistic Regression (LR) model, Random Forest (RF) model, Non-negative Matrix Factorization (NMF) model and Latent Dirichlet Allocation (LDA) model, etc.

[0230] In the embodiment of the present application, the triple pair includes the original word, the most similar word corresponding to the original word, and the similarity between the original word and the most similar word (ie, the similarity related to the original word).

[0231] 2. Calculate the mean of each similarity to obtain the database table similarity between the first database table and the second database table.

[0232] 3. When the library table similarity is greater than a preset threshold, it is determined that there is a semantically similar relationship between the first library table and the second library table.

[0233] Of course, when the library table similarity is not greater than the preset threshold, it is determined that there is no semantic similarity relationship between the first library table and the second library table.

[0234] 4. Sort the original words in descending order of similarity with the original words to obtain the original word sequence.

[0235] 5. Select the original words that are ranked in the first n positions in the original word sequence as the similarity labels of the first library table and the second library table.

[0236] It should be noted that n is a positive integer.

[0237] S136: The inclusion relationship between each work set, the inclusion relationship between the work set and the ETL task, the dependency relationship between each ETL task, the dependency relationship between each library table, the dependency relationship between each field, the inclusion relationship between the ETL task and the library table, the inclusion relationship between the library table and the field, and the semantic similarity relationship between each library table are saved in the preset library table as the data lineage relationship of the data warehouse.

[0238] Among them, the preset library table can specifically be a relational library table. Specifically, the fields of the relational library table can be designed as [object, object type, object source]. The so-called object type includes work set, ETL task, library table, field, and the so-called object source is used to record the data lineage relationship of the object.

[0239] It should be noted that the relational database table can also record the basic attributes of the object, including but not limited to: environment name, update time, maintainer and other information. The basic attributes of the object can generally be obtained by querying the code repository of the data warehouse.

[0240] In addition, relational tables can also record the similarity labels of each table to provide an effective basis for aggregate display analysis.

[0241] S137: Performing graph conversion on the data blood relationship recorded in the preset library table to obtain a relationship graph, and displaying the relationship graph to the user through a preset interface.

[0242] The specific implementation process of the graph conversion is common knowledge familiar to those skilled in the art. Specifically, it can be implemented using a graph data warehouse (such as neo4j). In the embodiment of the present application, each node in the relationship graph is each object in the preset library table, and the relationship between each node in the relationship graph is the data kinship relationship of the objects in the preset library table.

[0243] Based on the above process shown in S101-S137, this embodiment can achieve the following beneficial effects:

[0244] 1. It realizes the automatic analysis of data lineage relationship, avoids manual participation in the analysis process of data lineage relationship, saves a lot of manpower maintenance costs, and the analyzed data lineage relationship is more accurate and real-time.

[0245] 2. Implement a component parser algorithm based on Kettle, which solves the problem that the open source ETL tool Kettle lacks metadata management function for task content. It can be used in other Kettle-based data warehouse construction scenarios, not limited to a specific project.

[0246] 3. It realizes flexible data lineage relationship storage, storing lineage relationship information in the form of relational database tables and relationship graphs. It also supports two different query scenarios: table analysis and path display, without the need for separate development for the display scenario.

[0247] To sum up, the solution shown in this embodiment can help users quickly and effectively learn about the data lineage relationships contained in the data warehouse. Compared with the existing technology, there is no need for manual participation in the data lineage relationship analysis process, which saves a lot of manpower costs, and the data lineage relationships obtained through analysis are more accurate and real-time.

[0248] It should be noted that S101 mentioned in the above embodiment is an optional implementation of the data lineage processing method based on the data warehouse shown in the embodiment of the present application. In addition, S130 mentioned in the above embodiment is also an optional implementation of the data lineage processing method based on the data warehouse shown in the embodiment of the present application. To this end, the process shown in the above embodiment can be summarized as follows: Figure 2 The method shown.

[0249] like Figure 2 FIG. 1 is a flow chart of another data kinship processing method based on a data warehouse provided in an embodiment of the present application, comprising the following steps:

[0250] S201: Obtain the working set and ETL tasks contained in the data warehouse.

[0251] S202: using a preset regular expression, obtaining from the character information of the workset the inclusion relationship between the worksets and the inclusion relationship between the workset and the ETL task.

[0252] S203: Parse the character information of the work set to obtain the dependency relationship between each ETL task.

[0253] S204: Parse the ETL task to obtain various components included in the ETL task.

[0254] S205: parse each component to obtain each library table, dependency relationships between each library table, each field, and dependency relationships between each field contained in each component.

[0255] S206: Based on the dependency relationships between the library tables included in each component, the dependency relationships between the library tables in the data warehouse are constructed.

[0256] S207: Based on the dependency relationship between the fields included in each component, the dependency relationship between the fields in the data warehouse is constructed.

[0257] S208: Based on each library table included in the ETL task, generate an inclusion relationship between the ETL task and the library table.

[0258] S209: Acquire the fields included in each library table, and generate an inclusion relationship between the library table and the fields based on the fields included in each library table.

[0259] S210: Acquire a valid word set of each library table, and determine a semantic similarity relationship between each library table based on the valid word set of each library table.

[0260] To sum up, the solution shown in this embodiment can help users quickly and effectively learn about the data lineage relationships contained in the data warehouse. Compared with the existing technology, there is no need for manual participation in the data lineage relationship analysis process, which saves a lot of manpower costs, and the data lineage relationships obtained through analysis are more accurate and real-time.

[0261] Corresponding to the data lineage relationship processing method based on a data warehouse provided in the above-mentioned embodiment of the present application, the embodiment of the present application also provides a data lineage relationship processing device based on a data warehouse.

[0262] like Figure 3 FIG. 1 is a schematic diagram of the architecture of a data lineage relationship processing device based on a data warehouse provided in an embodiment of the present application, including:

[0263] The task acquisition unit 100 is used to acquire the work set and ETL tasks contained in the data warehouse.

[0264] Among them, the task acquisition unit 100 is specifically used to: obtain the working sets and ETL tasks contained in the data warehouse from a pre-constructed third information table; wherein the construction process of the third information table includes: obtaining various project files of the data warehouse; identifying the project files with the file suffix .kjb in each project file as working sets, and identifying the project files with the file suffix .ktr as ETL tasks; and constructing the third information table based on each working set and ETL task.

[0265] The regular analysis unit 200 is used to obtain the inclusion relationship between each work set and the inclusion relationship between the work set and the ETL task from the character information of the work set by using a preset regular expression.

[0266] Among them, the regular analysis unit 200 is specifically used to: obtain a string that matches the first regular expression from the character information of the workset; obtain a string that matches the second regular expression from the string that matches the first regular expression; obtain a string that matches the third regular expression from the character information of the workset; obtain a string that matches the fourth regular expression from the string that matches the second regular expression, and parse the string that matches the fourth regular expression to obtain other worksets included in the workset; obtain a string that matches the fifth regular expression from the string that matches the third regular expression, and parse the ETL tasks included in the workset from the string that matches the fifth regular expression; generate an inclusion relationship between each workset based on other worksets included in each workset; generate an inclusion relationship between a workset and an ETL task based on the ETL tasks included in each workset.

[0267] The work set parsing unit 300 is used to parse the character information of the work set to obtain the dependency relationship between each ETL task.

[0268] Among them, the work set parsing unit 300 is specifically used to: obtain a character string matching a first keyword from the character information of the work set; obtain a character string matching a second keyword from the character string matching the first keyword; parse the name of the source task and the name of the target task that depends on the source task from the character string matching the second keyword; obtain a character string matching a third keyword from the character information of the work set; parse each file path from the character string matching the third keyword; obtain a file path containing the name of the source task and a file path containing the name of the target task from each file path; query and obtain the ETL task corresponding to the source task based on the file path containing the name of the source task; query and obtain the ETL task corresponding to the target task based on the file path containing the name of the target task; generate the dependency relationship between each ETL task based on the dependency relationship between the source task and the target task, the ETL task corresponding to the source task, and the ETL task corresponding to the target task.

[0269] The task parsing unit 400 is used to parse the ETL task to obtain various components included in the ETL task.

[0270] Types of components include stored procedures and table output components.

[0271] The component parsing unit 500 is used to parse each component to obtain each library table contained in each component, the dependency relationship between each library table, each field, and the dependency relationship between each field.

[0272] Among them, the component parsing unit 500 is specifically used for: for each stored procedure included in the ETL task, obtaining the effective implementation logic of the stored procedure from the pre-constructed fourth information table, and marking the effective implementation logic as the logic to be parsed; wherein the construction process of the fourth information table includes: pre-acquiring each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the outline of the project logic; calling the preset SQL query statement to query from the outline to obtain the implementation logic of each stored procedure included in the project logic; performing data cleaning on the implementation logic of each stored procedure to obtain the effective implementation logic of each stored procedure; performing data format conversion on the effective implementation logic of each stored procedure so that the data format of the effective implementation logic of each effective stored procedure is the same; constructing the fourth information table based on the effective implementation logic of each stored procedure after the data format conversion; parsing the logic to be parsed to obtain each library table, the dependency relationship between each library table, each field, and the dependency relationship between each field contained in the stored process.

[0273] The component parsing unit 500 is specifically used for: obtaining each string matching the fourth keyword from the logic to be parsed; parsing each output table contained in the logic to be parsed from each string matching the fourth keyword; obtaining each string matching the fifth keyword from the logic to be parsed; parsing each library table contained in the logic to be parsed from each string matching the fifth keyword; using the library table whose table name is different from the table name of the output table in each library table as the input table contained in the logic to be parsed; pairing each output table contained in the logic to be parsed with each input table contained in the logic to be parsed, respectively, to obtain the dependency relationship between each library table contained in the storage process; obtaining each string matching the sixth keyword from the logic to be parsed; parsing each output field contained in the logic to be parsed from each string matching the sixth keyword; obtaining each string matching the seventh keyword from the logic to be parsed; parsing each candidate field contained in the logic to be parsed from each string matching the seventh keyword; for each candidate field, after determining the pre-constructed When a candidate field is recorded in an information table, the candidate field is used as the input field contained in the logic to be parsed; wherein, the construction process of the first information table includes: obtaining each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the data warehouse information of the project logic; calling the preset SQL query statement to query and obtain the metadata information table of the project logic from the data warehouse information; the metadata information table includes various meta-information; each meta-information includes a library table, the table name and description information of the library table, the field contained in the library table, the field name and description information of the field; data cleaning is performed on each meta-information to obtain each valid meta-information; data format conversion is performed on each valid meta-information so that the data format of each valid meta-information is the same; based on each valid meta-information after the data format conversion, the first information table is constructed; using the field name of the input field as the search basis, the field name search is performed on each output field to obtain the input field on which each output field depends; based on the input field on which each output field depends, the dependency relationship between each field is generated.

[0274] The component parsing unit 500 is specifically used for: for each table output component included in the ETL task, obtaining each string matching the eighth keyword from the table output component; parsing each string matching the eighth keyword to obtain each output table included in the table output component; obtaining each string matching the ninth keyword from the table output component; parsing each input table included in the table output component from each string matching the ninth keyword; pairing each output table included in the table output component with each input table included in the table output component to obtain the dependency relationship between each library table; obtaining each string matching the sixth keyword from the table output component; From each character string that matches the sixth keyword, parse and obtain each output field contained in the table output component; obtain each character string that matches the seventh keyword from the table output component; from each character string that matches the seventh keyword, parse and obtain each candidate field contained in the table output component; for each candidate field, when it is determined that the candidate field is recorded in the pre-constructed first information table, use the candidate field as the input field contained in the table output component; use the field name of the input field as the search basis, perform a field name search on each output field, and obtain the input field that each output field depends on; based on the input field that each output field depends on, generate the dependency relationship between the fields.

[0275] The first construction unit 600 is used to construct the dependency relationship between various library tables in the data warehouse based on the dependency relationship between various library tables included in each component.

[0276] The second construction unit 700 is used to construct the dependency relationship between various fields in the data warehouse based on the dependency relationship between various fields included in each component.

[0277] The first generating unit 800 is used to generate an inclusion relationship between the ETL task and the library table based on each library table included in the ETL task.

[0278] The second generating unit 900 is used to obtain the fields included in each library table, and generate the inclusion relationship between the library table and the fields based on the fields included in each library table.

[0279] The determination unit 1000 is used to obtain a valid word set of each library table, and determine the semantic similarity relationship between each library table based on the valid word set of each library table.

[0280] The determination unit 1000 is specifically used for: for each library table included in the ETL task, obtaining the table name and description information of the library table from the pre-constructed first information table; performing word segmentation on the table name and description information of the library table respectively to obtain a candidate word set of the library table; the candidate word set includes multiple candidate words; obtaining a synonym set corresponding to each candidate word from the pre-constructed second information table; the synonym set includes multiple synonyms; wherein the construction process of the second information table includes: obtaining each project logic of the data warehouse, and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the data warehouse information of the project logic; calling a preset SQL query statement to query and obtain the metadata information table of the project logic from the data warehouse information; the metadata information table includes various meta-information; each meta-information includes the library table, the fields contained in the library table, the field name and description information of the field; performing data cleaning on each meta-information to obtain each valid meta-information; performing semantic recognition on each valid meta-information to obtain each business noun, and configuring a corresponding synonym set; based on each business noun and the synonym set corresponding to each business noun, construct a second information table; from each candidate word, select a candidate word with a word type of noun and mark it as a valid word; from each synonym corresponding to each candidate word, select a synonym with a word type of verb and mark it as a valid word; based on each valid word, construct a valid word set of the library table; input the valid words in the first valid word set and the valid words in the second valid word set into the preset word similarity model to obtain each ternary pair output by the word similarity model; the first valid word set is the valid word set of the first library table; the second valid word set is the valid word set of the second library table; the first library table and the second library table are both library tables included in the ETL task; the ternary pair includes the original word, the most similar word corresponding to the original word, and the similarity related to the original word; the mean of each similarity is calculated to obtain the library table similarity between the first library table and the second library table; when the library table similarity is greater than the preset threshold, it is determined that there is a semantic similarity relationship between the first library table and the second library table.

[0281] The saving unit 1100 is used to save the inclusion relationship between various work sets, the inclusion relationship between work sets and ETL tasks, the dependency relationship between various ETL tasks, the dependency relationship between various library tables, the dependency relationship between various fields, the inclusion relationship between ETL tasks and library tables, the inclusion relationship between library tables and fields, and the semantic similarity relationship between various library tables as data lineage relationships in the data warehouse, into preset library tables.

[0282] The display unit 1200 is used to convert the data blood relationship recorded in the preset library table into a graph to obtain a relationship graph, and display the relationship graph to the user through a preset interface.

[0283] To sum up, the solution shown in this embodiment can help users quickly and effectively learn about the data lineage relationships contained in the data warehouse. Compared with the existing technology, there is no need for manual participation in the data lineage relationship analysis process, which saves a lot of manpower costs, and the data lineage relationships obtained through analysis are more accurate and real-time.

[0284] If the functions described in the method of the embodiment of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a storage medium readable by a computing device. Based on this understanding, the part of the embodiment of the present application that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computing device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories, random access memories, magnetic disks or optical disks.

[0285] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0286] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / tableoutput> < / entry> < / hop> < / hops>

Claims

1. A data kinship processing method based on a data warehouse, characterized in that: include: Get the working sets and ETL tasks contained in the data warehouse; Using a preset regular expression, the inclusion relationship between the work sets and the inclusion relationship between the work sets and the ETL task are obtained from the character information of the work sets; Parsing the character information of the work set to obtain the dependency relationship between each of the ETL tasks; Parsing the ETL task to obtain various components included in the ETL task; Parsing each of the components to obtain each library table contained in each component, dependencies between the library tables, each field, and dependencies between the fields; Based on the dependency relationship between the library tables contained in each component, construct the dependency relationship between the library tables in the data warehouse; Based on the dependency relationship between the fields contained in each component, construct the dependency relationship between the fields in the data warehouse; Based on each of the library tables included in the ETL task, generating an inclusion relationship between the ETL task and the library table; Acquire the fields included in each of the library tables, and based on the fields included in each of the library tables, generate an inclusion relationship between the library tables and the fields; A valid word set of each of the library tables is obtained, and based on the valid word set of each of the library tables, a semantic similarity relationship between each of the library tables is determined.

2. The method according to claim 1, characterized in that The obtaining of the working set and ETL tasks contained in the data warehouse includes: Obtain the working sets and ETL tasks contained in the data warehouse from the pre-built third information table; Among them, the construction process of the third information table includes: obtaining each project file of the data warehouse; identifying the project files with the file suffix .kjb in each of the project files as the working set, and identifying the project files with the file suffix .ktr as the ETL task; and constructing the third information table based on each of the working sets and the ETL tasks.

3. The method according to claim 1, characterized in that: The method of using a preset regular expression to obtain the inclusion relationship between the work sets and the inclusion relationship between the work sets and the ETL task from the character information of the work sets includes: Acquire a character string matching a first regular expression from the character information of the working set; From the strings that match the first regular expression, obtain the strings that match the second regular expression; Obtain the character string that matches the third regular expression from the character information of the working set; Acquire a string matching a fourth regular expression from the string matching the second regular expression, and parse the string matching the fourth regular expression to obtain other work sets included in the work set; Acquire a string matching a fifth regular expression from the string matching the third regular expression, and parse the string matching the fifth regular expression to obtain the ETL task included in the work set; Based on other work sets contained in each work set, generating an inclusion relationship between the work sets; Based on the ETL tasks included in each of the work sets, an inclusion relationship between the work sets and the ETL tasks is generated.

4. The method according to claim 1, characterized in that The character information of the work set is parsed to obtain the dependency relationship between each of the ETL tasks, including: Acquire a character string matching the first keyword from the character information of the work set; From the character strings that match the first keyword, obtain the character strings that match the second keyword; Parsing the string that matches the second keyword to obtain the name of the source task and the name of the target task that depends on the source task; Acquire a character string matching a third keyword from the character information of the work set; Parsing the character string that matches the third keyword to obtain each file path; Acquire a file path including the name of the source task and a file path including the name of the target task from each of the file paths; According to the file path containing the name of the source task, query and obtain the ETL task corresponding to the source task; According to the file path containing the name of the target task, query and obtain the ETL task corresponding to the target task; Based on the dependency relationship between the source task and the target task, the ETL task corresponding to the source task, and the ETL task corresponding to the target task, a dependency relationship between the ETL tasks is generated.

5. The method according to claim 1, characterized in that The types of components include stored procedures; The parsing of each component to obtain each library table contained in each component, the dependency relationship between each library table, each field, and the dependency relationship between each field includes: For each stored procedure included in the ETL task, obtaining the effective implementation logic of the stored procedure from a pre-constructed fourth information table, and marking the effective implementation logic as the logic to be parsed; Among them, the construction process of the fourth information table includes: pre-acquiring each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the outline of the project logic; calling the preset SQL query statement to query from the outline to obtain the implementation logic of each stored procedure contained in the project logic; performing data cleaning on the implementation logic of each stored procedure to obtain the effective implementation logic of each stored procedure; performing data format conversion on the effective implementation logic of each stored procedure so that the data format of the effective implementation logic of each effective stored procedure is the same; based on the effective implementation logic of each stored procedure after the data format conversion, constructing the fourth information table; The logic to be analyzed is analyzed to obtain various library tables, dependency relationships between the library tables, various fields, and dependency relationships between the fields contained in the storage process.

6. The method according to claim 5, characterized in that The step of parsing the logic to be parsed to obtain various library tables, dependencies between the library tables, various fields, and dependencies between the fields contained in the storage process includes: Acquire each character string matching the fourth keyword from the logic to be parsed; Parsing each character string that matches the fourth keyword to obtain each output table included in the logic to be parsed; Acquire each character string matching the fifth keyword from the logic to be parsed; Parsing each character string that matches the fifth keyword to obtain each library table included in the logic to be parsed; Using the library tables whose table names are different from the table names of the output tables as the input tables included in the logic to be parsed; Pair each output table included in the logic to be parsed with each input table included in the logic to be parsed, to obtain the dependency relationship between the library tables included in the storage process; Acquire each character string matching the sixth keyword from the logic to be parsed; Parsing each character string that matches the sixth keyword to obtain each output field included in the logic to be parsed; Acquire each character string matching the seventh keyword from the logic to be parsed; Parsing each character string that matches the seventh keyword to obtain each candidate field included in the to-be-parsed logic; For each of the candidate fields, when it is determined that the candidate field is recorded in the pre-constructed first information table, the candidate field is used as an input field included in the logic to be parsed; Among them, the construction process of the first information table includes: obtaining each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the data warehouse information of the project logic; calling the preset SQL query statement to query and obtain the metadata information table of the project logic from the data warehouse information; the metadata information table includes various meta information; each meta information includes a library table, the table name and description information of the library table, the field contained in the library table, the field name and description information of the field; performing data cleaning on each meta information to obtain each valid meta information; performing data format conversion on each valid meta information so that the data format of each valid meta information is the same; based on each valid meta information after the data format conversion, constructing the first information table; Using the field name of the input field as a search basis, searching the field name of each output field to obtain the input field that each output field depends on; Based on the input fields that each of the output fields depends on, a dependency relationship between the fields is generated.

7. The method according to claim 1, characterized in that The types of components include table output components; The parsing of each component to obtain each library table contained in each component, the dependency relationship between each library table, each field, and the dependency relationship between each field includes: For each table output component included in the ETL task, obtaining each character string matching the eighth keyword from the table output component; Parsing each character string matched with the eighth keyword to obtain each output table included in the table output component; Acquire each character string matching the ninth keyword from the table output component; Parsing each character string matched with the ninth keyword to obtain each input table included in the table output component; Pairing each output table included in the table output component with each input table included in the table output component to obtain dependency relationships between various library tables; Acquire each character string matching the sixth keyword from the table output component; Parsing each character string matching the sixth keyword to obtain each output field included in the table output component; Acquire each character string matching the seventh keyword from the table output component; Parsing each character string matched with the seventh keyword to obtain each candidate field included in the table output component; For each of the candidate fields, when it is determined that the candidate field is recorded in the pre-constructed first information table, the candidate field is used as an input field included in the table output component; Using the field name of the input field as a search basis, searching the field name of each output field to obtain the input field that each output field depends on; Based on the input fields that each of the output fields depends on, a dependency relationship between the fields is generated.

8. The method according to claim 1, characterized in that: The obtaining of valid word sets of each of the library tables, and determining semantic similarity relationships between each of the library tables based on the valid word sets of each of the library tables, includes: For each of the library tables included in the ETL task, obtain the table name and description information of the library table from a pre-built first information table; Segmenting the table name and description information of the library table respectively to obtain a candidate word set of the library table; the candidate word set includes multiple candidate words; Acquire a synonym set corresponding to each of the candidate words from a pre-constructed second information table; the synonym set includes a plurality of synonyms; The construction process of the second information table includes: obtaining each project logic of the data warehouse and the configuration file of each project logic; for each project logic, parsing the configuration file of the project logic to obtain the data warehouse information of the project logic; calling a preset SQL query statement to query and obtain the metadata information table of the project logic from the data warehouse information; the metadata information table includes various meta information; each meta information includes a library table, a field contained in the library table, a field name and description information of the field; performing data cleaning on each meta information to obtain each valid meta information; performing semantic recognition on each valid meta information to obtain each business noun, and configuring a synonym set corresponding to each business noun; based on each business noun and a synonym set corresponding to each business noun, constructing the second information table; From each of the candidate words, select a candidate word whose word type is a noun and mark it as a valid word; From the synonyms corresponding to each of the candidate words, a synonym of a verb type is selected and marked as a valid word; Based on each of the valid words, construct a valid word set in the library table; Input valid words in the first valid word set and valid words in the second valid word set into a preset word similarity model to obtain each triple pair output by the word similarity model; the first valid word set is a valid word set of the first library table; the second valid word set is a valid word set of the second library table; the first library table and the second library table are both library tables included in the ETL task; the triple pair includes an original word, a most similar word corresponding to the original word, and a similarity related to the original word; Calculating the mean of each of the similarities to obtain the library table similarity between the first library table and the second library table; When the library table similarity is greater than a preset threshold, it is determined that there is a semantically similar relationship between the first library table and the second library table.

9. The method according to claim 1, characterized in that: Also includes: The inclusion relationship between the various working sets, the inclusion relationship between the working sets and the ETL tasks, the dependency relationship between the various ETL tasks, the dependency relationship between the various library tables, the dependency relationship between the various fields, the inclusion relationship between the ETL tasks and the library tables, the inclusion relationship between the library tables and the fields, and the semantic similarity relationship between the various library tables are saved in the preset library table as the data lineage relationship of the data warehouse; The data blood relationship recorded in the preset library table is converted into a graph to obtain a relationship graph, and the relationship graph is displayed to the user through a preset interface.

10. A data kinship processing device based on a data warehouse, characterized in that: include: A task acquisition unit is used to acquire the working set and ETL tasks contained in the data warehouse; A regular analysis unit, used for obtaining the inclusion relationship between each of the work sets and the inclusion relationship between the work set and the ETL task from the character information of the work set by using a preset regular expression; A work set parsing unit, used to parse the character information of the work set to obtain the dependency relationship between each of the ETL tasks; A task parsing unit, used to parse the ETL task to obtain various components included in the ETL task; A component parsing unit, used for parsing each of the components to obtain each library table contained in each component, the dependency relationship between each of the library tables, each field, and the dependency relationship between each of the fields; A first construction unit, configured to construct a dependency relationship between the library tables in the data warehouse based on the dependency relationship between the library tables contained in each component; A second construction unit, configured to construct a dependency relationship between the fields in the data warehouse based on the dependency relationship between the fields included in each component; A first generating unit, configured to generate an inclusion relationship between the ETL task and the library table based on each of the library tables included in the ETL task; A second generating unit, used for acquiring the fields included in each of the library tables, and generating an inclusion relationship between the library table and the fields based on the fields included in each of the library tables; The determination unit is used to obtain the valid word sets of each of the library tables, and determine the semantic similarity relationship between each of the library tables based on the valid word sets of each of the library tables.

Images