Data blood relationship management method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data consanguinity management method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a collection request which comprises a collection target address; performing data acquisition according to the acquisition destination address to obtain acquired data, and performing blood relationship information processing on the acquired data to obtain a data source table corresponding to the acquired data; when the file in the preset format is monitored, generating a data source table and a data conversion table corresponding to the file in the preset format according to the file in the preset format; obtaining a query request, and generating a data consumption table according to the query request; and generating a data blood relationship according to the data source table, the data conversion table and the data consumption table. Therefore, the data blood relationship can be automatically generated, the data blood relationship can be managed, manual labeling and recording are not needed, and the phenomenon of incomplete or inaccurate information caused by manual labeling and recording during blood relationship management is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data lineage management method, a data lineage management device, an electronic device and a computer-readable storage medium. Background Art

[0002] Nowadays, the development of many enterprises relies on massive data support, but as the amount of data continues to expand, enterprises' demand for data governance is also increasing. Data lineage analysis can help enterprises govern data and reveal the complex relationships and dependencies between data. Traditional data lineage analysis often relies on manual labeling and recording, which is not only inefficient, but may also lead to incomplete or inaccurate information. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data lineage management method, a data lineage management device, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problems, an embodiment of the present invention discloses a data lineage management method, the method comprising:

[0005] Obtaining a collection request, wherein the collection request includes a collection target address;

[0006] Collecting data according to the collection destination address to obtain collected data, and processing the collected data with lineage information to obtain a data source table corresponding to the collected data;

[0007] When a preset format file is detected, a data source table and a data conversion table corresponding to the preset format file are generated according to the preset format file;

[0008] Obtaining a query request, and generating a data consumption table according to the query request;

[0009] A data lineage relationship is generated according to the data source table, the data conversion table and the data consumption table.

[0010] Optionally, the blood relationship information processing of the collected data to obtain a data source table corresponding to the collected data includes:

[0011] Processing the collected data for blood relationship information to obtain processed blood relationship information;

[0012] Fields that conform to a preset field format are extracted from the processed blood relationship information, and the fields are written into a preset data source table to generate a data source table corresponding to the collected data.

[0013] Optionally, when a preset format file is monitored, generating a data source table and a data conversion table corresponding to the preset format file according to the preset format file includes:

[0014] When a preset format file is detected, the preset format file is parsed to obtain a configuration file of the preset format file;

[0015] Obtaining a conversion type and a data name from the configuration file;

[0016] According to the conversion type and the data name, a data source table and a data conversion table corresponding to the preset format file are generated.

[0017] Optionally, obtaining the query request and generating a data consumption table according to the query request includes:

[0018] Obtaining a query request, parsing a query statement in the query request, and obtaining a name of the queried data;

[0019] A data consumption table is generated according to the data name.

[0020] Optionally, generating the data lineage relationship according to the data source table, the data conversion table and the data consumption table includes:

[0021] Obtain data identifiers in the data source table, the data conversion table, and the data consumption table;

[0022] Determining identical data identifiers from the data identifiers;

[0023] A data lineage relationship is generated based on the same data identifier.

[0024] Optionally, extracting fields conforming to a preset field format from the processed blood relationship information, and writing the fields into a preset data source table to generate a data source table corresponding to the collected data includes:

[0025] Extracting fields corresponding to data identification, data name, data type, source information and storage information respectively from the processed blood relationship information;

[0026] The fields corresponding to the data identification, data name, data type, source information and storage information are written into a preset data source table to generate a data source table corresponding to the collected data.

[0027] Optionally, generating a data consumption table according to the data name includes:

[0028] According to the data name, determining a data identifier corresponding to the data name from the data source table;

[0029] Obtaining consumer information corresponding to the query request;

[0030] A data consumption table is generated according to the data identifier corresponding to the data name and the consumer information corresponding to the query request.

[0031] The embodiment of the present invention further discloses a data lineage management device, the device comprising:

[0032] An acquisition module, used to acquire an acquisition request, wherein the acquisition request includes an acquisition target address;

[0033] A processing module, used to collect data according to the collection destination address to obtain collected data, and to process the collected data for lineage information to obtain a data source table corresponding to the collected data;

[0034] A first generating module, for generating a data source table and a data conversion table corresponding to a preset format file according to the preset format file when a preset format file is monitored;

[0035] A second generating module is used to obtain a query request and generate a data consumption table according to the query request;

[0036] The third generation module is used to generate data kinship according to the data source table, the data conversion table and the data consumption table.

[0037] Optionally, the processing module includes:

[0038] The processing submodule is used to process the collected data to obtain the processed bloodline information;

[0039] The first generating submodule is used to extract fields that conform to a preset field format from the processed blood relationship information, and write the fields into a preset data source table to generate a data source table corresponding to the collected data.

[0040] Optionally, the first generating module includes:

[0041] A first parsing submodule is used to parse the preset format file when a preset format file is monitored, and obtain a configuration file of the preset format file;

[0042] A first acquisition submodule, used for acquiring a conversion type and a data name from the configuration file;

[0043] The second generating submodule is used to generate a data source table and a data conversion table corresponding to the preset format file according to the conversion type and the data name.

[0044] Optionally, the second generating module includes:

[0045] The second parsing submodule is used to obtain a query request, parse the query statement in the query request, and obtain the queried data name;

[0046] The third generating submodule is used to generate a data consumption table according to the data name.

[0047] Optionally, the third generating module includes:

[0048] A second acquisition submodule is used to acquire data identifiers in the data source table, the data conversion table, and the data consumption table;

[0049] A determination submodule, used to determine identical data identifiers from the data identifiers;

[0050] The fourth generating submodule is used to generate data lineage relationship according to the same data identifier.

[0051] Optionally, the first generating submodule includes:

[0052] An extraction unit, used to extract fields corresponding to data identification, data name, data type, source information and storage information from the processed blood relationship information;

[0053] The first generating unit is used to write the fields corresponding to the data identification, data name, data type, source information and storage information respectively into a preset data source table to generate a data source table corresponding to the collected data.

[0054] Optionally, the third generation submodule includes:

[0055] a determination unit, configured to determine, according to the data name, a data identifier corresponding to the data name from the data source table;

[0056] An acquisition unit, used to acquire consumer information corresponding to the query request;

[0057] The second generating unit is used to generate a data consumption table according to the data identifier corresponding to the data name and the consumer information corresponding to the query request.

[0058] The present invention also discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data lineage management method when executing the computer program.

[0059] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data lineage management method as described above are implemented.

[0060] The embodiments of the present invention include the following advantages:

[0061] In an embodiment of the present invention, a collection request is obtained, and then data collection is performed according to the collection destination address in the collection request to obtain the collected data, and the collected data is processed for lineage information to obtain a data source table corresponding to the collected data. Then, when a preset format file is monitored, a data source table and a data conversion table corresponding to the preset format file are generated according to the preset format file, and then a query request is obtained, and a data consumption table is generated according to the query request. Finally, data lineage relationships are generated according to the data source table, the data conversion table and the data consumption table, thereby automatically generating data lineage relationships and managing data lineage without manual labeling and recording, thereby avoiding the phenomenon of incomplete or inaccurate information caused by manual labeling and recording when lineage management is performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flowchart of the steps of a data lineage management method provided by an embodiment of the present invention;

[0063] Figure 2 is a flow chart of data collection steps provided by an embodiment of the present invention;

[0064] Figure 3 It is a flow chart of steps for generating a data source table and a data conversion table provided by an embodiment of the present invention;

[0065] Figure 4 is a flow chart of steps for generating a data consumption table provided by an embodiment of the present invention;

[0066] Figure 5 is a schematic diagram of a data kinship relationship provided by an embodiment of the present invention;

[0067] Figure 6 It is a structural block diagram of a data lineage management device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] Traditional data lineage analysis management often relies on manual labeling and recording, which is not only inefficient, but also may lead to incomplete or inaccurate information. In order to solve the above technical problems, the present invention proposes a data lineage management method, obtains a collection request, and then collects data according to the collection destination address in the collection request to obtain the collected data, and processes the lineage information of the collected data to obtain a data source table corresponding to the collected data, and then when a preset format file is monitored, according to the preset format file, a data source table and a data conversion table corresponding to the preset format file are generated, and then a query request is obtained, and a data consumption table is generated according to the query request. Finally, data lineage relationships are generated according to the data source table, the data conversion table and the data consumption table, so that data lineage relationships can be automatically generated and data lineage can be managed without manual labeling and recording, avoiding the phenomenon of incomplete or inaccurate information caused by manual labeling and recording when lineage management is performed.

[0070] Reference Figure 1 , shows a flow chart of the steps of a data lineage management method provided by an embodiment of the present invention, and the method may specifically include the following steps:

[0071] Step 101: Obtain a collection request, wherein the collection request includes a collection target address.

[0072] In an embodiment of the present invention, the data lineage management method can be applied to a data lineage management platform, which can include: a monitoring system, a data lineage collection system, a data lineage storage system, and a data lineage display system. In the present invention, the monitoring system can send a collection request to the data lineage collection system, and the data lineage collection system can obtain the collection request, which can include a collection target address.

[0073] The monitoring system of the present invention can be Prometheus, which is a system monitoring and alarm toolkit. The main features of Prometheus are as follows: Prometheus uses a time series data model to identify and query data through key-value pairs (labels). This multidimensional data model enables Prometheus to flexibly process and query complex monitoring data. Prometheus provides a powerful query language (PromQL) for real-time query and analysis of time series data. PromQL supports complex query, aggregation and data conversion operations. Prometheus supports automatic discovery of targets (such as services, containers, etc.) and automatically collects monitoring data for these targets. Prometheus can dynamically discover and monitor targets through service discovery mechanisms (such as Kubernetes, Consul, etc.). Prometheus supports distributed deployment and high availability configuration to ensure that the monitoring system can still work normally in the event of a failure. Prometheus uses local storage (TSDB) to store time series data, and also supports remote storage (such as InfluxDB, OpenTSDB, etc.). Prometheus supports integration with various tools and platforms, such as Grafana (for visualization), Alertmanager (for alert management), Kubernetes (for service discovery), etc.

[0074] Step 102, data is collected according to the collection destination address to obtain collected data, and the collected data is processed according to the lineage information to obtain a data source table corresponding to the collected data.

[0075] In an embodiment of the present invention, after obtaining a collection request, the data lineage collection system may collect data according to the collection destination address to obtain collected data, and then perform lineage processing on the collected data to obtain a data source table corresponding to the collected data.

[0076] Specifically, Figure 2As shown, a flow chart of the steps of data collection provided by an embodiment of the present invention is shown. Prometheus sends a collection request to the data lineage collection system. The collection request includes the collection target address. The data lineage collection system sends the collection request to the collection target according to the collection target address. After receiving the collection request, the collection target responds to the collection request to generate a collection request response, and sends the collection request response to the data lineage collection system. After receiving the collection request response, the data lineage collection system can perform data collection to obtain the collected data, and generate a data source table according to the collected data. After generating the data source table, the data lineage collection system can send the corresponding request to Prometheus, and at the same time, the data source table can be stored in the data lineage storage system. When receiving a query request for the data source table, the data lineage storage system can send the data source table to the data lineage display system, so that the data lineage display system displays the data source table.

[0077] In one embodiment, bloodline information processing is performed on the collected data to obtain a data source table corresponding to the collected data, which may include: bloodline information processing is performed on the collected data to obtain processed bloodline information; fields that conform to a preset field format are extracted from the processed bloodline information, and the fields are written into a preset data source table to generate a data source table corresponding to the collected data.

[0078] In the present invention, the data lineage collection system can process the lineage information of the collected data to obtain the processed lineage information, and then extract the fields that meet the preset field format from the processed lineage information, the fields in the preset format are the same as the data information required by the preset data source table, and then write the fields in the preset field format into the preset data source table, thereby generating a data source table corresponding to the collected data. The preset format fields of the present invention can be fields corresponding to data identification, data name, data type, source information and storage information respectively.

[0079] In one embodiment, fields that conform to a preset field format are extracted from processed bloodline information, and the fields are written into a preset data source table to generate a data source table corresponding to the collected data, including: extracting fields corresponding to the data identifier, data name, data type, source information, and storage information, respectively, from the processed bloodline information; writing the fields corresponding to the data identifier, data name, data type, source information, and storage information, respectively, into the preset data source table to generate a data source table corresponding to the collected data.

[0080] Specifically, the data lineage collection system can extract fields corresponding to data identification, data name, data type, source information and storage information, and write the extracted fields into a preset data source table to generate a data source table. The following Table 1 is a generated data source table.

[0081] Field Name Data Identification Data name Data Types Source Information Storage Information 1 Indicator name 1 Collection indicators 5.5.5.5 1.1.1.1 2 Indicator name 2 Collection indicators 5.5.5.5,6.6.6.6 1.1.1.1 3 Indicator name 3 Collection indicators 6.6.6.6 1.1.1.1

[0082] Table 1

[0083] In the present invention, the source information may use IP or host name to represent the data source address information, and the storage information may be the address information of Prometheus. The specific writing method of the source information and the storage information may be set according to actual needs, and the present invention does not make specific limitations here.

[0084] A specific example of obtaining the above Table 1 is that the address information of Prometheus is 1.1.1.1. Through the lineage data collection system, data is collected for targets 1 and 2 respectively. The address information of target 1 is 5.5.5.5, and the address information of target 2 is 6.6.6.6. Target 1 returns the data of indicator name 1 and indicator name 2, and target 2 returns the data of indicator name 2 and indicator name 3. Then, the data source table of Table 1 above can be generated according to the data returned by targets 1 and 2.

[0085] Step 103: when a preset format file is monitored, a data source table and a data conversion table corresponding to the preset format file are generated according to the preset format file.

[0086] In an embodiment of the present invention, the data lineage collection system can monitor the preset file of Prometheus in real time. The preset file can be a rule file. After monitoring the rule file, a data source table and a data conversion table corresponding to the rule file can be generated according to the rule file.

[0087] In Prometheus, rule files (also called rule files) are configuration files used to define alert rules and recording rules. Rule files allow users to define complex queries and conditions to trigger alerts or record new time series data when monitoring data meets specific conditions. Rule files are an important part of the Prometheus monitoring system and are used to automate alerts and data processing.

[0088] In one embodiment, when a preset format file is monitored, a data source table and a data conversion table corresponding to the preset format file are generated according to the preset format file, including: when a preset format file is monitored, the preset format file is parsed to obtain a configuration file of the preset format file; a conversion type and a data name are obtained from the configuration file; and a data source table and a data conversion table corresponding to the preset format file are generated according to the conversion type and the data name. In the present invention, the data conversion table may include a conversion identifier, a data source identifier, a data destination identifier, and a conversion type. Among them, the data source identifier corresponds to the data identifier, and the data destination identifier corresponds to the data identifier.

[0089] Specifically, when the data lineage collection system monitors the rule file, it can parse the rule file and obtain the configuration file of the rule file. The general format of the rule file is as follows:

[0090] -name: calculation group

[0091] rules:

[0092] -record: new indicator data name

[0093] expr: calculation method

[0094] In the rule file, expr will record which indicator data is used for calculation, and record is the name of the calculated data. The configuration file corresponding to the rule file is generally as follows:

[0095] -name: calculation group

[0096] rules:

[0097] -record: indicator data name 2

[0098] expr:sum(index data name 1)

[0099] Based on the above Table 1, the configuration file of the rule file is as follows:

[0100] -name: calculation group

[0101] rules:

[0102] -record: indicator data 10

[0103] expr:sum by(instance)(index data 1{})and count by(instance)(index data 2{})

[0104] -record: indicator data 11

[0105] expr:avg by(instance)(index data3{})

[0106] According to the above configuration file, the conversion type in the configuration file can be obtained as record calculation, the data names are indicator data 10 and data indicator 11, indicator data 10 is calculated by indicator data 1 and indicator data 2, and indicator data 11 is calculated by indicator data 3. Based on this, the data source table in Table 2 below and the data conversion table in Table 3 below can be generated.

[0107]

[0108] Table 2

[0109] Field Name Conversion ID Data source identifier Data target identifier Conversion Type 1 1,2 4 record calculation 2 3 5 record calculation

[0110] Table 3

[0111] like Figure 3 As shown, a flowchart of the steps of generating a data source table and a data conversion table provided by an embodiment of the present invention is shown. The data lineage collection system monitors the rule file in Prometheus. When the rule file is monitored, the data source table and the data conversion table corresponding to the rule file are generated according to the rule file, and the data source table and the data conversion table are sent to the data lineage storage system. When the data lineage storage system receives a query request for the data source table and the data conversion table, the data source table and the data conversion table are sent to the data lineage display system, so that the data lineage display system displays the data source table and the data conversion table.

[0112] It should be noted that, in actual applications, the records of conversion relationships in the data conversion table can be multi-level, with many-to-many relationship records between sources and targets, thereby realizing relationship records of data lineage.

[0113] Step 104: Obtain a query request, and generate a data consumption table according to the query request.

[0114] In the embodiment of the present invention, when the data lineage collection system obtains a query request from a data query party, it can generate a data consumption table according to the query request.

[0115] like Figure 4As shown, a flow chart of the steps of generating a data consumption table provided by an embodiment of the present invention is shown. The data query party can send a query request to the data lineage collection system. After receiving the query request, the data lineage collection system can send the query request to Prometheus. After receiving the query request, Prometheus can generate a query response and send the query response to the data lineage collection system. After receiving the query response, the data lineage collection system can generate a data consumption table and send the data consumption table to the data lineage storage system. At the same time, the query response will be sent to the data query party. The data lineage storage system can send the data consumption table to the data lineage display module so that the data lineage display system can display the data consumption table.

[0116] In one embodiment, obtaining a query request and generating a data consumption table according to the query request includes: obtaining the query request, parsing the query statement in the query request to obtain the queried data name; and generating the data consumption table according to the data name.

[0117] Specifically, after receiving the query request, the data lineage collection system can parse the query statement in the query request to extract the data name to be queried. For example, if the queried data name is indicator data 1 and indicator data 11, then the corresponding data consumption table can be generated based on indicator data 1 and indicator data 11.

[0118] In one embodiment, generating a data consumption table based on a data name may include: determining a data identifier corresponding to the data name from a data source table based on the data name; obtaining consumer information corresponding to the query request; and generating a data consumption table based on the data identifier corresponding to the data name and the consumer information corresponding to the query request.

[0119] Specifically, a query request is obtained, and the query statement in the query request can be as follows:

[0120] rate(index data 1{}[5m])

[0121] count(index data 11{})

[0122] By analyzing the above query statement, we can get that the queried data names are indicator data 1 and indicator data 11. Then, according to indicator data 1 and indicator data 11, we can determine that the data identifier corresponding to indicator data 1 is 1, and the data identifier corresponding to indicator data 11 is 5. At the same time, we can also obtain the IP of the data query party, that is, we can obtain the consumer information, for example, the consumer information is 9.9.9.9, and then generate the data consumption table in Table 4 according to the data identifier corresponding to the data name and the consumer information corresponding to the query request:

[0123] Field Name Consumption data identification Consumer information 1 9.9.9.9 5 9.9.9.9

[0124] Table 4

[0125] Step 105: Generate data lineage relationships based on the data source table, the data conversion table, and the data consumption table.

[0126] In the embodiment of the present invention, after the data source table, the data conversion table and the data consumption table are generated, the data blood relationship can be generated according to the data source table, the data conversion table and the data consumption table.

[0127] In one embodiment, data lineage relationships are generated based on a data source table, a data conversion table, and a data consumption table, including: obtaining data identifiers in the data source table, the data conversion table, and the data consumption table; determining identical data identifiers from the data identifiers; and generating data lineage relationships based on identical data identifiers.

[0128] Specifically, we continue to explain with the above example. According to the data source table and the data conversion table, we can know that data identifier 4 is converted from data identifier 1 and data identifier 2, and data identifier 5 is converted from data identifier 3, that is, indicator data 10 is calculated from indicator data 1 and indicator data 2, and indicator data 11 is calculated from indicator data 3. From the data consumption table, we can know that the queried data identifiers are 1 and 5, that is, the queried data are indicator data 1 and indicator data 11. Therefore, we can get the following: Figure 5 The data lineage relationship, Figure 5 A schematic diagram showing a data lineage relationship.

[0129] Finally, the present invention generates data lineage relationships according to the data source table, the data conversion table and the data consumption table, thereby achieving complete non-intrusion to Prometheus, automatically collating and collecting data lineage relationships in massive indicator data, and realizing flexible and automated data lineage management.

[0130] In an embodiment of the present invention, a collection request is obtained, the collection request includes a collection target address; data is collected according to the collection target address to obtain the collected data, and the collected data is processed with lineage information to obtain a data source table corresponding to the collected data; when a preset format file is monitored, a data source table and a data conversion table corresponding to the preset format file are generated according to the preset format file; a query request is obtained, and a data consumption table is generated according to the query request; data lineage relationships are generated according to the data source table, the data conversion table, and the data consumption table. In this way, data lineage relationships can be automatically generated, and data lineage can be managed without manual labeling and recording, thereby avoiding the phenomenon of incomplete or inaccurate information caused by manual labeling and recording when lineage management is performed.

[0131] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0132] Reference Figure 6 , shows a structural block diagram of a data lineage management device provided by an embodiment of the present invention, which may specifically include the following modules:

[0133] The acquisition module 601 is used to acquire a collection request, wherein the collection request includes a collection target address;

[0134] Processing module 602, used to collect data according to the collection destination address to obtain collected data, and process the collected data with lineage information to obtain a data source table corresponding to the collected data;

[0135] The first generating module 603 is used to generate a data source table and a data conversion table corresponding to the preset format file according to the preset format file when the preset format file is monitored;

[0136] The second generating module 604 is used to obtain a query request and generate a data consumption table according to the query request;

[0137] The third generating module 605 is used to generate data kinship according to the data source table, the data conversion table and the data consumption table.

[0138] In one embodiment, the processing module 602 includes:

[0139] The processing submodule is used to process the collected data to obtain the processed bloodline information;

[0140] The first generating submodule is used to extract fields that conform to a preset field format from the processed blood relationship information, and write the fields into a preset data source table to generate a data source table corresponding to the collected data.

[0141] In one embodiment, the first generating module 603 includes:

[0142] A first parsing submodule is used to parse the preset format file when a preset format file is monitored, and obtain a configuration file of the preset format file;

[0143] A first acquisition submodule, used for acquiring a conversion type and a data name from the configuration file;

[0144] The second generating submodule is used to generate a data source table and a data conversion table corresponding to the preset format file according to the conversion type and the data name.

[0145] In one embodiment, the second generating module 604 includes:

[0146] The second parsing submodule is used to obtain a query request, parse the query statement in the query request, and obtain the queried data name;

[0147] The third generating submodule is used to generate a data consumption table according to the data name.

[0148] In one embodiment, the third generation module 605 includes:

[0149] A second acquisition submodule is used to acquire data identifiers in the data source table, the data conversion table, and the data consumption table;

[0150] A determination submodule, used to determine identical data identifiers from the data identifiers;

[0151] The fourth generating submodule is used to generate data lineage relationship according to the same data identifier.

[0152] In one embodiment, the first generating submodule includes:

[0153] An extraction unit, used to extract fields corresponding to the field name, data identifier, data name, data type, source information and storage information respectively from the processed blood relationship information;

[0154] The first generating unit is used to write the fields corresponding to the field name, data identifier, data name, data type, source information and storage information respectively into a preset data source table to generate a data source table corresponding to the collected data.

[0155] In one embodiment, the third generation submodule includes:

[0156] a determination unit, configured to determine, according to the data name, a data identifier corresponding to the data name from the data source table;

[0157] An acquisition unit, used to acquire consumer information corresponding to the query request;

[0158] The second generating unit is used to generate a data consumption table according to the data identifier corresponding to the data name and the consumer information corresponding to the query request.

[0159] In an embodiment of the present invention, the acquisition module is used to obtain a collection request, and the collection request includes a collection target address; the processing module is used to collect data according to the collection target address, obtain the collected data, and process the collected data with lineage information to obtain a data source table corresponding to the collected data; the first generation module is used to generate a data source table and a data conversion table corresponding to the preset format file according to the preset format file when a preset format file is monitored; the second generation module is used to obtain a query request and generate a data consumption table according to the query request; the third generation module is used to generate data lineage relationships according to the data source table, the data conversion table, and the data consumption table. In this way, data lineage relationships can be automatically generated, and data lineage can be managed without manual labeling and recording, thereby avoiding the phenomenon of incomplete or inaccurate information caused by manual labeling and recording when lineage management is performed.

[0160] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0161] An embodiment of the present invention further provides an electronic device, including:

[0162] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data lineage management method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0163] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data lineage management method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0164] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0165] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0166] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0169] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0170] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0171] The data lineage management method, data lineage management device, electronic device and computer-readable storage medium provided by the present invention are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data lineage management method, characterized in that: The method comprises: Obtaining a collection request, wherein the collection request includes a collection target address; Collecting data according to the collection destination address to obtain collected data, and processing the collected data with lineage information to obtain a data source table corresponding to the collected data; When a preset format file is detected, a data source table and a data conversion table corresponding to the preset format file are generated according to the preset format file; Obtaining a query request, and generating a data consumption table according to the query request; A data lineage relationship is generated according to the data source table, the data conversion table and the data consumption table.

2. The data lineage management method according to claim 1, characterized in that: The blood relationship information processing of the collected data to obtain a data source table corresponding to the collected data includes: Processing the collected data for blood relationship information to obtain processed blood relationship information; Fields that conform to a preset field format are extracted from the processed blood relationship information, and the fields are written into a preset data source table to generate a data source table corresponding to the collected data.

3. The data lineage management method according to claim 1, characterized in that: When a preset format file is monitored, generating a data source table and a data conversion table corresponding to the preset format file according to the preset format file, including: When a preset format file is detected, the preset format file is parsed to obtain a configuration file of the preset format file; Obtaining a conversion type and a data name from the configuration file; According to the conversion type and the data name, a data source table and a data conversion table corresponding to the preset format file are generated.

4. The data lineage management method according to claim 1, characterized in that: The obtaining of the query request and generating a data consumption table according to the query request include: Obtaining a query request, parsing a query statement in the query request, and obtaining a name of the queried data; A data consumption table is generated according to the data name.

5. The data lineage management method according to claim 1, characterized in that: The generating of data kinship relationships according to the data source table, the data conversion table and the data consumption table includes: Obtain data identifiers in the data source table, the data conversion table, and the data consumption table; Determining identical data identifiers from the data identifiers; A data lineage relationship is generated based on the same data identifier.

6. The data lineage management method according to claim 2, characterized in that: The step of extracting fields conforming to a preset field format from the processed blood relationship information and writing the fields into a preset data source table to generate a data source table corresponding to the collected data includes: Extracting fields corresponding to data identification, data name, data type, source information and storage information respectively from the processed blood relationship information; The fields corresponding to the data identification, data name, data type, source information and storage information are written into a preset data source table to generate a data source table corresponding to the collected data.

7. The data lineage management method according to claim 4, characterized in that: The step of generating a data consumption table according to the data name includes: According to the data name, determining a data identifier corresponding to the data name from the data source table; Obtaining consumer information corresponding to the query request; A data consumption table is generated according to the data identifier corresponding to the data name and the consumer information corresponding to the query request.

8. A data lineage management device, characterized in that: The device comprises: An acquisition module, used to acquire an acquisition request, wherein the acquisition request includes an acquisition target address; A processing module, used to collect data according to the collection destination address to obtain collected data, and to process the collected data for lineage information to obtain a data source table corresponding to the collected data; A first generating module, for generating a data source table and a data conversion table corresponding to a preset format file according to the preset format file when a preset format file is monitored; A second generating module is used to obtain a query request and generate a data consumption table according to the query request; The third generation module is used to generate data kinship according to the data source table, the data conversion table and the data consumption table.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data lineage management method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data lineage management method according to any one of claims 1 to 7 are implemented.

Images