Data lineage management method, device, electronic device and storage medium

Abstract

The present application proposes a data lineage management method, device, electronic device and storage medium, wherein the data lineage management method includes: obtaining a release table; obtaining a local calculation table; according to a fusion calculation strategy, fusing the release table with the calculation table to generate a new release table, wherein the new release table includes version information; registering with a data-oriented architecture DOA registration center according to the new release table to obtain a new access path from the DOA registration center; determining a first accounting information table corresponding to the new access path according to the new access path; updating the first accounting information table according to the new release table, version information and new access path; sending the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage. Thus, data lineage management can be achieved, and data nodes do not need to have data lineage calculation capabilities, thereby effectively ensuring the integrity of data lineage information.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data lineage management method, device, electronic device and storage medium. Background Art

[0002] Data lineage refers to the relationship formed between data during the process of data generation, processing, circulation and extinction, which is similar to the blood relationship in human society.

[0003] The current data lineage is based on the collection, analysis and processing of data lineage in a proprietary domain, that is, data management nodes (with lineage analysis capabilities) collect data from nodes in the data circulation process, data development nodes and other processing logic to form data lineage relationships.

[0004] In the related technology, due to the separation of data and control information, nodes where data flows, data development nodes, etc. are required to obtain the address information of the data management node and send the data processing logic to the data management node; or the data management node obtains the data processing logic information of each node from the nodes where data flows, the data development node, and performs summary analysis to calculate the data lineage. Summary of the invention

[0005] The embodiments of the present application provide a data lineage management method, device, electronic device and storage medium, which can realize data lineage management, and the data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0006] The first aspect of the present application proposes a data lineage management method, including: obtaining a publication table, wherein the publication table includes a local publication table or an external publication table; obtaining a local calculation table; according to a fusion calculation strategy, fusing the publication table with the calculation table to generate a new publication table, wherein the new publication table includes version information; registering with a data-oriented architecture DOA registration center according to the new publication table to obtain a new access path from the DOA registration center; determining a first accounting information table corresponding to the new access path according to the new access path; updating the first accounting information table according to the new publication table, the version information and the new access path; sending the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage.

[0007] In addition, the data lineage management method according to the above embodiment of the present application may also have the following additional technical features:

[0008] In one embodiment of the present application, the publishing table includes an external publishing table, and obtaining the publishing table includes: obtaining an external access path, and obtaining the external publishing table according to the external access path, wherein the external publishing table is published by a data node.

[0009] In one embodiment of the present application, the publication table includes a local publication table, and obtaining the publication table includes: receiving a data resource publication instruction, wherein the data resource publication instruction includes version information of the data resource and a publication table of the data resource; and using the publication table of the data resource as the local publication table.

[0010] In one embodiment of the present application, after obtaining the publication table, it also includes: registering with the data-oriented architecture DOA registration center according to the data resource publication instruction to obtain the access path of the data resource from the DOA registration center; determining a second accounting information table corresponding to the access path according to the access path; updating the second accounting information table according to the local publication table, the version information of the data resource and the access path; sending the updated second accounting information table and the structured query language SQL statement corresponding to the second accounting information table to the blockchain for storage.

[0011] In one embodiment of the present application, the data lineage management method further includes: publishing the local publishing table according to the access path.

[0012] In one embodiment of the present application, the updating of the second accounting information table according to the local publication table, the version information of the data resource and the access path includes: obtaining the publication time of the local publication table; writing the table name of the local publication table, the version information of the data resource, the access path and the publication time into the second accounting information table; wherein, if the local publication table is created locally, NA is written into the second accounting information table as the table name of the local publication table.

[0013] In one embodiment of the present application, there are multiple calculation tables, and according to the fusion calculation strategy, the publishing table is fused with the calculation table to generate a new publishing table, including: according to the fusion calculation strategy, the multiple calculation tables are fused with the publishing table in sequence to generate the new publishing table.

[0014] In one embodiment of the present application, the data lineage management method further includes: publishing the new publishing table according to the new access path.

[0015] In one embodiment of the present application, the updating of the first accounting information table according to the newly published table, the version information and the new access path includes: obtaining the new publication time of the newly published table; writing the table name of the newly published table, the version information, the new access path and the new publication time into the first accounting information table.

[0016] The second aspect embodiment of the present application proposes a data lineage management method, including: obtaining an accounting information table and an SQL statement corresponding to the accounting information table from a blockchain; performing data lineage analysis based on the accounting information table and the SQL statement to generate a data lineage analysis result.

[0017] In addition, the data lineage management method according to the above embodiment of the present application may also have the following additional technical features:

[0018] In one embodiment of the present application, the data lineage analysis is performed according to the accounting information table and the SQL statement to generate a data lineage analysis result, including: iteratively analyzing each row of data in the accounting information table according to the SQL statement to obtain the lineage path information corresponding to each row of data; performing data lineage analysis on the lineage path information corresponding to each row of data to generate a data lineage analysis result.

[0019] The third aspect embodiment of the present application proposes a data lineage management device, including: a first acquisition module, used to obtain a publication table, wherein the publication table includes a local publication table or an external publication table; a second acquisition module, used to obtain a local calculation table; a fusion calculation module, used to fuse the publication table with the calculation table according to a fusion calculation strategy to generate a new publication table, wherein the new publication table includes version information; a third acquisition module, used to register with a data-oriented architecture DOA registration center according to the new publication table to obtain a new access path from the DOA registration center; a determination module, used to determine a first accounting information table corresponding to the new access path according to the new access path; an update module, used to update the first accounting information table according to the new publication table, the version information and the new access path; a sending module, used to send the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage.

[0020] The fourth aspect embodiment of the present application proposes a data lineage management device, including: an acquisition module, used to obtain an accounting information table and an SQL statement corresponding to the accounting information table from a blockchain; an analysis module, used to analyze data lineage based on the accounting information table and the SQL statement to generate a data lineage analysis result.

[0021] The fifth aspect of the present application proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data lineage management method as described above is implemented.

[0022] The electronic device of the embodiment of the present application can implement data lineage management by executing a computer program stored in a memory through a processor, and the data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0023] The sixth aspect of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data lineage management method as described in the first aspect of the present application is implemented.

[0024] The computer-readable storage medium of the embodiment of the present application can realize the management of data lineage by storing computer programs and being executed by the processor, and the data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0025] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart of a data lineage management method according to an embodiment of the present application;

[0028] Figure 2 It is a flowchart of a data lineage management method according to another embodiment of the present application;

[0029] Figure 3 It is a flowchart of a data lineage management method according to another embodiment of the present application;

[0030] Figure 4 It is a flowchart of a data lineage management method according to another embodiment of the present application;

[0031] Figure 5 It is a flowchart of a data lineage management method according to another embodiment of the present application;

[0032] Figure 6 It is a flowchart of a data lineage management method according to another embodiment of the present application;

[0033] Figure 7 A schematic diagram of the architecture of a data lineage management method according to an embodiment of the present application;

[0034] Figure 8 It is a block diagram of a data lineage management device according to an embodiment of the present application;

[0035] Fig. 9 is a block diagram of a data lineage management device according to another embodiment of the present application; and

[0036] Fig.10 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0038] The following describes the data lineage management method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0039] The data lineage management method provided in the embodiment of the present application may be executed by an electronic device, which may be a computer or a server, etc., without any limitation herein.

[0040] In the embodiment of the present application, a processing component, a storage component and a drive component may be provided in the electronic device. Optionally, the drive component and the processing component may be integrated, the storage component may store an operating system, an application program or other program modules, and the processing component implements the data lineage management method provided in the embodiment of the present application by executing the application program stored in the storage component.

[0041] Figure 1 A flowchart of a data lineage management method provided according to an embodiment of the present application.

[0042] The data lineage management method of the embodiment of the present application can also be executed by the data lineage management device provided by the embodiment of the present application. The device can be configured in an electronic device to obtain a publication table and a local calculation table, and fuse the publication table with the calculation table according to a fusion calculation strategy to generate a new publication table, wherein the new publication table includes version information, and then registers with the data-oriented architecture DOA registration center according to the new publication table to obtain a new access path from the DOA registration center, and determines the first accounting information table corresponding to the new access path according to the new access path, and updates the first accounting information table according to the new publication table, version information and new access path, and then sends the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage, thereby realizing data lineage management, and data nodes do not need to have data lineage calculation capabilities, thereby effectively ensuring the integrity of data lineage information.

[0043] As a possible scenario, the data lineage management method of the embodiment of the present application may also be executed by a data node, wherein an electronic device (for example, a server) may be the execution subject of the data node, and the specific functions of the data node are implemented by the electronic device.

[0044] like Figure 1 As shown, the data lineage management method may include:

[0045] Step 101, obtaining a publishing table, wherein the publishing table includes a local publishing table or an external publishing table. There may be multiple publishing tables.

[0046] To clearly illustrate the previous embodiment, in one embodiment of the present application, the publishing table may include an external publishing table, and obtaining the publishing table may include obtaining an external access path, and obtaining the external publishing table according to the external access path, wherein the external publishing table is published by a data node.

[0047] In the embodiment of the present application, the external access path mentioned above may be input by a user, or may be automatically acquired by a data node in response to a related event.

[0048] Specifically, after obtaining the external access path, the data node (i.e., the agent process of the data node) can obtain the external publishing table (i.e., data resource) corresponding to the external access path based on the external access path, wherein the external publishing table can be published by other data nodes, without any limitation here.

[0049] To clearly illustrate the previous embodiment, in another embodiment of the present application, the publishing table may include a local publishing table, and obtaining the publishing table may include receiving a data resource publishing instruction, wherein the data resource publishing instruction includes version information of the data resource and a publishing table of the data resource, and the publishing table of the data resource is used as the local publishing table.

[0050] In the embodiment of the present application, the above-mentioned data resource release table can be pre-created by relevant personnel. When relevant personnel need to release data resources, they can first create a release table for the data resource and specify the version information of the data resource, wherein the version information can also be directly written into the release table, without any limitation here. Then, the relevant client (platform) can generate a data resource release instruction according to the release table and the version information, and send it to the data node (for example, the corresponding data node). The data node receives and responds.

[0051] Specifically, relevant personnel can create a data resource publishing instruction on the corresponding client, and send the data resource publishing instruction to the data node, which will receive and respond to it.

[0052] As a possible scenario, the data node can detect data resource publishing events in real time through the relevant API (Application Programming Interface), and after detecting the data resource publishing event, automatically create a data resource publishing instruction and issue the data resource publishing instruction to receive and respond to the data resource publishing instruction.

[0053] Furthermore, after receiving the data resource publishing instruction, the above data node may first parse the data resource publishing instruction to obtain the publishing table of the data resource in the data resource publishing instruction, and may use the publishing table as a local publishing table for subsequent operations.

[0054] In order to further clarify the above embodiment, in one embodiment of the present application, Figure 2 As shown, after obtaining the publishing table (ie, the local publishing table), the following steps may also be included:

[0055] Step 201: register with the data-oriented architecture DOA registration center according to the data resource release instruction to obtain the access path of the data resource from the DOA registration center.

[0056] It should be noted that the access path of the data resource described in this embodiment may be an access path after the data resource (ie, the publishing table (local publishing table)) is published, and the access path may be generated (provided) by the DOA registration center.

[0057] Among them, the data-oriented architecture DOA registration center can register various types of data and generalized data to form a logical data resource pool to facilitate application access to data. Its functions involve but are not limited to: data registration information definition, data attribute information, data classification, metadata standards, metadata classification, registration methods for different types of data, data indexing, metadata indexing, data retrieval, generalized data pattern recognition, distributed deployment, data registration content on-demand adaptive mechanism, data generation automatic registration mechanism, historical data registration and management, etc. The data-oriented architecture DOA registration center can provide (generate) access paths to data resources corresponding to the version information during registration according to the version information.

[0058] Specifically, after receiving the data resource publishing instruction, the data node (i.e., the agent process of the data node) may first parse the data resource publishing instruction to obtain the version information of the data resource and the publishing table of the data resource (i.e., the local publishing table) in the data resource publishing instruction. Then, the data node may register with the data-oriented architecture DOA registration center according to the data resource publishing instruction to obtain the access path of the data resource from the DOA registration center. Among them, the version information of the data resource is an auxiliary information. When registering, it is the registration data table name (e.g., the publishing table name) and the metadata information contained. The data version number (i.e., the version information) is one of the information of the metadata.

[0059] Furthermore, in one embodiment of the present application, the data lineage management method may also include publishing the local publishing table according to the access path.

[0060] Specifically, after obtaining the access path to the data resource, the data node may publish the data resource to the address corresponding to the access path, that is, publish the above-mentioned local publishing table to the address corresponding to the access path, thereby realizing external publishing of the data resource.

[0061] Step 202: Determine a second accounting information table corresponding to the access path according to the access path. It should be noted that the second accounting information table described in this embodiment may be a local table, for example, an accounting information table of a data node.

[0062] As a possible scenario, the second accounting information table may also be an external table, and the second accounting information table may be confirmed and obtained based on the access path.

[0063] Step 203, update the second accounting information table according to the local publishing table, version information of the data resource and the access path.

[0064] In order to clearly explain the above embodiment, in one embodiment of the present application, Figure 3As shown, updating the second accounting information table according to the local publishing table, the version information of the data resource and the access path may include:

[0065] Step 301, obtaining the publishing time of the local publishing table.

[0066] In an embodiment of the present application, when a data node publishes a local publication table according to an access path, it may record the publication time of the local publication table and may keep (or temporarily store) it in its own storage space for easy retrieval when needed.

[0067] Step 302, write the table name of the local publishing table, version information of the data resource, access path and publishing time into the second accounting information table. If the local publishing table is created locally, NA is written into the second accounting information table as the table name of the local publishing table, where NA can represent "empty".

[0068] Specifically, after obtaining the above-mentioned second accounting information table, the data node can obtain the publishing time of the above-mentioned local publishing table from its own storage space, and write the table name, version information of the data resource, access path and publishing time of the above-mentioned local publishing table into the second accounting information table, that is, write the table name, version information of the data resource, access path and publishing time of the above-mentioned local publishing table into the second accounting information table as a set of row information. It should be noted that while writing the table name, version information, access path and publishing time of the above-mentioned local publishing table as a set of row information into the second accounting information table, its input table information can also be written into the second accounting information table together.

[0069] The second accounting information table may be shown in Table 1 below:

[0070]

[0071] Table 1

[0072] It should be noted that the column "Whether to involve external view" in Table 1 described in this embodiment refers to whether the table corresponding to the data table name in this row (for example, the local publishing table) is published externally. If yes, "Y" can be written in this row, and if not, "N" can be written in this row. That is, it can be understood that the table corresponding to the data table name is a resource obtained from outside the data node.

[0073] Step 204: Send the updated second accounting information table and the structured query language SQL statement corresponding to the second accounting information table to the blockchain for storage.

[0074] It should be noted that the SQL statement described in this embodiment may be a structured query language, wherein the structured query language is a database query and programming language used to access data and query, update and manage relational database systems (e.g., tables in a database system). The SQL statement may include DDL (Data Definition Language, database schema definition language) statements and DML (Data Manipulation Language, data manipulation language) statements.

[0075] Among them, the SQL statement corresponding to the above-mentioned second accounting information table can be stored in the log of the database corresponding to the second accounting information.

[0076] Specifically, after the data node writes the table name of the above-mentioned local publishing table, the version information of the data resource, the access path and the publishing time into the second accounting information table, the second accounting information table (i.e., the second accounting information table after writing) and the SQL statement corresponding to the second accounting information table can be sent to the blockchain. After receiving the second accounting information table and the SQL statement corresponding to the second accounting information table, the blockchain (e.g., the blockchain client) can store them in the block body of the blockchain. This is to facilitate the subsequent data management node to obtain the accounting information table (i.e., the second accounting information table) and the SQL statement corresponding to the accounting information table (i.e., the second accounting information table) from the block body to analyze the data lineage.

[0077] It should be noted that a blockchain network may include multiple data nodes, wherein the execution of the above Figure 2 and Figure 3 The data node of the data lineage management method of the embodiment may be one of the multiple data nodes, and the data node may be a data node in the blockchain network that receives data resource publishing instructions.

[0078] In an embodiment of the present application, a data node may receive a data resource publishing instruction, wherein the data resource publishing instruction includes the version information of the data resource and the publishing table of the data resource (i.e., the local publishing table), and then register with the data-oriented architecture DOA registration center according to the data resource publishing instruction to obtain the access path of the data resource from the DOA registration center, and determine the second accounting information table corresponding to the access path according to the access path, and update the second accounting information table according to the publishing table (i.e., the local publishing table), the version information of the data resource and the access path, and finally send the updated second accounting information table and the structured query language SQL statement corresponding to the second accounting information table to the blockchain for storage. Thus, data lineage management can be achieved, and data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0079] Step 102: Obtain a local calculation table, wherein there may be multiple calculation tables.

[0080] Specifically, after obtaining the above-mentioned publishing table (for example, a local publishing table or an external publishing table), the data node may obtain a plurality of calculation tables stored locally.

[0081] Step 103: According to the fusion calculation strategy, the release table and the calculation table are fused and calculated to generate a new release table, wherein the new release table includes version information.

[0082] To clearly explain the previous embodiment, in one embodiment of the present application, there may be multiple calculation tables, and according to a fusion calculation strategy, the publishing table and the calculation table are fused together to generate a new publishing table, which may include fusing the multiple calculation tables with the publishing table in sequence according to the fusion calculation strategy to generate a new publishing table.

[0083] It should be noted that the fusion computing strategy described in this embodiment can be calibrated according to actual conditions, and the fusion computing strategy can be pre-stored in the storage space of the data node to facilitate subsequent calls and use.

[0084] Specifically, after obtaining the above-mentioned publishing table, the data node can obtain multiple calculation tables and fusion calculation strategies from its own storage space, and then according to the fusion calculation strategy, the multiple calculation tables can be fused with the publishing table in turn to generate a new publishing table.

[0085] It should be noted that the publishing table described in this embodiment can be merged with the local table (local calculation table) or with other external tables (external calculation tables) when merged locally (data node) to finally generate a new publishing table.

[0086] Step 104: register with the data-oriented architecture DOA registration center according to the new publishing table to obtain a new access path from the DOA registration center.

[0087] It should be noted that the new access path described in this embodiment may be an access path after the new publication table (new data resource) is published, and the new access path may be generated (provided) by the DOA registration center.

[0088] Among them, the data-oriented architecture DOA registration center can register various types of data and generalized data to form a logical data resource pool to facilitate application access to data. Its functions involve but are not limited to: data registration information definition, data attribute information, data classification, metadata standards, metadata classification, registration methods for different types of data, data indexing, metadata indexing, data retrieval, generalized data pattern recognition, distributed deployment, data registration content on-demand adaptive mechanism, data generation automatic registration mechanism, historical data registration and management, etc. The data-oriented architecture DOA registration center can provide (generate) the access path corresponding to the version information (i.e., the above-mentioned new access path) during the registration process according to the version information.

[0089] Specifically, after generating a new release table, the data node (i.e., the agent process of the data node) can first obtain the version information from the new release table, and then the data node can register with the data-oriented architecture DOA registration center according to the new release table to obtain a new access path (i.e., the access path of the new release table (new data resource)) from the DOA registration center. The above version information is an auxiliary information. When registering, it is the registration data table name (e.g., the new release table name) and the metadata information contained therein. The data version number (i.e., version information) is one of the information of the metadata.

[0090] Furthermore, in one embodiment of the present application, the data lineage management method may also include publishing a new publishing table according to the new access path.

[0091] Specifically, after obtaining the new access path, the data node may publish the new publishing table to the address corresponding to the new access path, that is, publish the data resources corresponding to the new publishing table to the address corresponding to the new access path, thereby realizing the external release of new data resources.

[0092] Step 105: Determine a first accounting information table corresponding to the new access path according to the new access path. It should be noted that the first accounting information table described in this embodiment may be a local table, for example, an accounting information table of a data node.

[0093] As a possible scenario, the first accounting information table mentioned above may also be an external table, and the accounting information table can be confirmed and obtained based on the new access path mentioned above.

[0094] It should be noted that the first accounting information table described in this embodiment and the second accounting information table described in the above embodiment may be the same table or different tables, and no limitation is made here.

[0095] Step 106, updating the first accounting information table according to the new release table, version information and new access path.

[0096] In order to clearly explain the above embodiment, in one embodiment of the present application, Figure 4 As shown, updating the first accounting information table according to the new release table, version information and new access path may include:

[0097] Step 401, obtaining the new publishing time of the new publishing table.

[0098] In an embodiment of the present application, when a data node publishes a new publication table according to a new access path, it can record the new publication time of the new publication table and keep (or temporarily store) it in its own storage space for easy retrieval when needed.

[0099] Step 402, write the table name, version information, new access path and new publishing time of the newly published table into the first accounting information table.

[0100] Specifically, after obtaining the above-mentioned first accounting information table, the data node can obtain the new release time of the above-mentioned newly released table from its own storage space, and write the table name, version information, new access path and new release time of the above-mentioned newly released table into the first accounting information table, that is, write the table name, version information, new access path and new release time of the above-mentioned newly released table into the first accounting information table as one or more groups of row information, wherein the input table (data source table) information can also be written into the first accounting information table at the same time. It should be noted that the first accounting information table described in this embodiment may include multiple rows of information.

[0101] The first accounting information table may be shown in Table 2 below:

[0102]

[0103] Table 2

[0104] It should be noted that the column "Whether it involves external communication" in Table 2 described in this embodiment refers to whether the table corresponding to the data table name in this row (for example, the newly released table) is released to the outside. If yes, "Y" can be written in this row, and if not, "N" can be written in this row. That is, it can be understood that the table corresponding to the data table name is a resource (for example, a released table) obtained from outside the data node.

[0105] Step 107: Send the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage.

[0106] It should be noted that the SQL statement described in this embodiment may be a structured query language, wherein the structured query language is a database query and programming language used to access data and query, update and manage relational database systems (e.g., tables in a database system). The SQL statement may include DDL statements and DML statements. The SQL statement corresponding to the first accounting information table may be stored in the log of the database corresponding to the first accounting information.

[0107] Specifically, after the data node writes the table name, version information, new access path and new release time of the newly released table into the first accounting information table, the first accounting information table (i.e., the first accounting information table after writing) and the SQL statement corresponding to the first accounting information table can be sent to the blockchain. After receiving the first accounting information table and the SQL statement corresponding to the first accounting information table, the blockchain (e.g., the blockchain client) can store them in the block body of the blockchain. This allows the subsequent data management node to obtain the accounting information table (e.g., the first accounting information table) and the SQL statement corresponding to the accounting information table (e.g., the first accounting information table) from the block body to perform data lineage analysis.

[0108] It should be noted that a blockchain network may include multiple data nodes, wherein the execution of the above Figures 1 to 4 The data node of the data lineage management method of the embodiment may be one or more of the multiple data nodes. If there are multiple data nodes, the multiple data nodes may execute the aforementioned Figures 1 to 4 A data lineage management method according to an embodiment.

[0109] The data lineage management method of the embodiment of the present application first obtains a release table, wherein the release table includes a local release table or an external release table, obtains a local calculation table, and fuses the release table with the calculation table according to a fusion calculation strategy to generate a new release table, wherein the new release table includes version information, and then registers with the data-oriented architecture DOA registration center according to the new release table to obtain a new access path from the DOA registration center, and determines the first accounting information table corresponding to the new access path according to the new access path, and updates the first accounting information table according to the new release table, version information and new access path, and finally sends the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage. In this way, data lineage management can be achieved, and data nodes do not need to have data lineage calculation capabilities, thereby effectively ensuring the integrity of data lineage information.

[0110] Figure 5 The present invention is a flowchart of another data lineage management method provided according to an embodiment of the present application.

[0111] The data lineage management method of the embodiment of the present application can also be executed by the data lineage management device provided by the embodiment of the present application. The device can be configured in an electronic device to obtain the accounting information table and the SQL statement corresponding to the accounting information table from the blockchain, and perform data lineage analysis based on the accounting information table and the SQL statement to generate data lineage analysis results, thereby realizing data lineage management, and data nodes do not need to have data lineage calculation capabilities, thereby effectively ensuring the integrity of data lineage information.

[0112] As a possible scenario, the data lineage management method of the embodiment of the present application can also be executed by a data management node, wherein an electronic device (for example, a server) can be the execution subject of the data management node and the specific functions of the data management node can be implemented by the electronic device.

[0113] like Figure 5 As shown, the data lineage management method may include:

[0114] Step 501, obtain the accounting information table and the SQL statement corresponding to the accounting information table from the blockchain.

[0115] Specifically, when data lineage analysis is required, the data management node can obtain the accounting information table and the SQL statement corresponding to the accounting information table from the block body of the blockchain (i.e., the blockchain client). It should be noted that the accounting information table described in this embodiment can be multiple, and no limitation is made here.

[0116] Step 502, performing data lineage analysis based on the accounting information table and the SQL statement to generate data lineage analysis results.

[0117] In the embodiment of the present application, data lineage analysis can be performed according to the preset accounting information table algorithm, accounting information table and SQL statement to generate data lineage analysis results. Among them, the preset accounting information table algorithm can be calibrated according to actual conditions.

[0118] Specifically, the data management node can query and analyze the input access path information and data table name on the accounting information table based on the preset accounting information table algorithm and the data resources (access path information and data table name (i.e., published table name)) that need to be analyzed, and then iteratively query and analyze the corresponding input access path information and data table name according to the corresponding access path information and data table name until the input table on all access paths is NA. Confirm the initial source of the data. Generate data lineage link information, i.e., data lineage analysis results, based on the access path information and data table name collected through analysis, as well as the corresponding DDL statements and DML statements.

[0119] In order to clearly explain the above embodiment, in one embodiment of the present application, Figure 6As shown, data lineage analysis is performed based on the accounting information table and SQL statements to generate data lineage analysis results, which may include:

[0120] Step 601, iteratively analyze each row of data in the accounting information table according to the SQL statement to obtain the bloodline path information corresponding to each row of data.

[0121] Step 602, performing data lineage analysis on the lineage path information corresponding to each row of data to generate a data lineage analysis result.

[0122] Specifically, after obtaining the accounting information table and the SQL statement corresponding to the accounting information table, the data management node can perform iterative analysis on each row of data in the accounting information table based on the preset accounting information table algorithm and the SQL statement to obtain the lineage path information corresponding to each row of data. The data management node can then perform data lineage analysis on the lineage path information corresponding to each row of data to generate a data lineage analysis result.

[0123] Further, in the embodiment of the present application, relevant personnel can regularly update the data in the data table (i.e., the release table), and without involving changes in the data table structure, according to the granularity of data lineage analysis, if it is necessary to regularly analyze the data lineage circulation data in the data lineage, the data node will update the accounting information table when incrementally updating the data (only involving DML statements), carrying the data resource data volume and release time. If it is not necessary to regularly analyze the data lineage circulation data, it is not necessary to update the accounting information table.

[0124] In an embodiment of the present application, by recording the data processing logic (i.e., the accounting information table and the SQL statement corresponding to the accounting information table) in the blockchain, the data node does not need to have the data lineage calculation capability, but only needs to publish the relevant data processing logic, and the data management node analyzes the data processing logic to generate data lineage.

[0125] The data lineage management method of the embodiment of the present application obtains the accounting information table and the SQL statement corresponding to the accounting information table from the blockchain, and performs data lineage analysis based on the accounting information table and the SQL statement to generate a data lineage analysis result, thereby realizing data lineage management, and the data nodes do not need to have data lineage calculation capabilities, thereby effectively ensuring the integrity of the data lineage information.

[0126] In order to enable those skilled in the art to more clearly understand the data lineage management method provided in the embodiments of the present application, Figure 7 FIG. 1 is a schematic diagram of the structure of a data lineage management method according to an embodiment of the present application. Figure 7As shown, the architecture may include a DOA registration center and a blockchain network, wherein the blockchain network may include multiple data nodes (i.e., a first data node, a second data node, and a third data node) and a data management node. The first data node, the second data node, and the third data node may sequentially execute the aforementioned Figures 1 to 4 The data lineage management method of the embodiment, the data management node can execute the aforementioned Figure 5 and Figure 6 The data lineage management method of the embodiment can thus achieve data lineage management, and the data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0127] Figure 8 It is a block diagram of a data lineage management device according to an embodiment of the present application.

[0128] The data lineage management device of the embodiment of the present application can be configured in an electronic device (for example, a server) to obtain a publication table and a local calculation table, and fuse the publication table with the calculation table according to a fusion calculation strategy to generate a new publication table, wherein the new publication table includes version information, and then register with the data-oriented architecture DOA registration center according to the new publication table to obtain a new access path from the DOA registration center, and determine the first accounting information table corresponding to the new access path according to the new access path, and update the first accounting information table according to the new publication table, version information and new access path, and then send the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage, so as to realize data lineage management, and the data nodes do not need to have data lineage calculation capabilities, thereby effectively ensuring the integrity of the data lineage information.

[0129] like Figure 8 As shown, the data lineage management device 800 may include: a first acquisition module 810, a second acquisition module 820, a fusion calculation module 830, a third acquisition module 840, a determination module 850, an update module 860 and a sending module 870.

[0130] The first acquisition module 810 is used to acquire a publishing table, wherein the publishing table includes a local publishing table or an external publishing table.

[0131] The second acquisition module 820 is used to acquire a local calculation table.

[0132] The fusion calculation module 830 is used to perform fusion calculation on the release table and the calculation table according to the fusion calculation strategy to generate a new release table, wherein the new release table includes version information.

[0133] The third acquisition module 840 is used to register with the data-oriented architecture DOA registration center according to the new publishing table to obtain a new access path from the DOA registration center.

[0134] The determination module 850 is used to determine the first accounting information table corresponding to the new access path according to the new access path.

[0135] The update module 860 is used to update the first accounting information table according to the new release table, version information and new access path.

[0136] The sending module 870 is used to send the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage.

[0137] In one embodiment of the present application, there are multiple calculation tables, and the fusion calculation module 930 is specifically used to: according to the fusion calculation strategy, sequentially fuse the multiple calculation tables with the publishing table to generate a new publishing table.

[0138] In one embodiment of the present application, the data lineage management device 800 may further include a publishing module, wherein the publishing module is used to publish a new publishing table according to a new access path.

[0139] In one embodiment of the present application, the update module 860 is specifically used to: obtain the new release time of the newly released table; write the table name, version information, new access path and new release time of the newly released table into the first accounting information table.

[0140] It should be noted that the above Figures 1 to 4 The explanation of the data lineage management method embodiment is also applicable to the data lineage management device of this embodiment, and will not be repeated here.

[0141] The data lineage management device of the embodiment of the present application first obtains the release table through the first acquisition module, wherein the release table includes a local release table or an external release table, and obtains the local calculation table through the second acquisition module, and the release table and the calculation table are fused and calculated according to the fusion calculation strategy by the fusion calculation module to generate a new release table, wherein the new release table includes version information, and then the third acquisition module is used to register with the data-oriented architecture DOA registration center according to the new release table to obtain a new access path from the DOA registration center, and the first accounting information table corresponding to the new access path is determined by the determination module according to the new access path, and the first accounting information table is updated according to the new release table, version information and new access path by the update module, and finally the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table are sent to the blockchain for storage by the sending module. In this way, the management of data lineage can be realized, and the data node does not need to have the data lineage calculation capability, so that the integrity of the data lineage information can be effectively guaranteed.

[0142] Fig. 9 It is a block diagram of a data lineage management device according to another embodiment of the present application.

[0143] The data lineage management device of the embodiment of the present application can be configured in an electronic device (for example, a server) to obtain the accounting information table and the SQL statement corresponding to the accounting information table from the blockchain, and perform data lineage analysis based on the accounting information table and the SQL statement to generate data lineage analysis results, thereby realizing data lineage management, and data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0144] like Fig. 9 As shown, the data lineage management device 900 may include: an acquisition module 910 and an analysis module 920 .

[0145] Among them, the acquisition module 910 is used to obtain the accounting information table and the SQL statement corresponding to the accounting information table from the blockchain.

[0146] The analysis module 920 is used to perform data lineage analysis based on the accounting information table and SQL statements to generate data lineage analysis results.

[0147] In one embodiment of the present application, the analysis module 920 is specifically used to: iteratively analyze each row of data in the accounting information table according to the SQL statement to obtain the lineage path information corresponding to each row of data; perform data lineage analysis on the lineage path information corresponding to each row of data to generate a data lineage analysis result.

[0148] It should be noted that the above Figures 5 to 7The explanation of the data lineage management method embodiment is also applicable to the data lineage management device of this embodiment, and will not be repeated here.

[0149] The data lineage management device of the embodiment of the present application obtains the accounting information table and the SQL statement corresponding to the accounting information table from the blockchain through the acquisition module, and performs data lineage analysis based on the accounting information table and the SQL statement through the analysis module to generate data lineage analysis results. In this way, data lineage management can be achieved, and data nodes do not need to have data lineage calculation capabilities, thereby effectively ensuring the integrity of data lineage information.

[0150] In order to implement the above embodiment, Fig.10 As shown, the present application also proposes an electronic device 1000, including a memory 1010, a processor 1020, and a computer program stored in the memory 1010 and executable on the processor 1020, and the processor 1020 executes the program to implement the data lineage management method proposed in the aforementioned embodiment of the present application.

[0151] The electronic device of the embodiment of the present application can implement data lineage management by executing a computer program stored in a memory through a processor, and the data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0152] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. The program is executed by a processor to implement the data lineage management method proposed in the above embodiments of the present application.

[0153] The computer-readable storage medium of the embodiment of the present application can realize the management of data lineage by storing computer programs and being executed by the processor, and the data nodes do not need to have data lineage computing capabilities, thereby effectively ensuring the integrity of data lineage information.

[0154] In the description of this specification, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0155] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0156] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A data lineage management method, characterized in that: include: The data node obtains an external access path, obtains an external publishing table published by other data nodes according to the external access path, or receives a data resource publishing instruction, the data resource publishing instruction includes a publishing table of the data resource, uses the publishing table of the data resource as a local publishing table, and uses the local publishing table or the external publishing table as a publishing table; After obtaining the publishing table, the data node also obtains a local calculation table stored locally; According to the fusion calculation strategy, the release table and the calculation table are fused and calculated to generate a new release table, wherein the new release table includes version information; Registering with a data-oriented architecture (DOA) registration center according to the new publication table to obtain a new access path from the DOA registration center; Determine a first accounting information table corresponding to the new access path according to the new access path; updating the first accounting information table according to the new publishing table, the version information and the new access path; The updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table are sent to the blockchain for storage, so that the data management node obtains the structured query language SQL statement from the blockchain.

2. The data lineage management method according to claim 1, characterized in that: The data resource publishing instruction also includes version information of the data resource.

3. The data lineage management method according to claim 2, characterized in that: After obtaining the publication table, the method further includes: Registering with a data-oriented architecture (DOA) registration center according to the data resource publishing instruction to obtain an access path to the data resource from the DOA registration center; Determine a second accounting information table corresponding to the access path according to the access path; updating the second accounting information table according to the local publishing table, the version information of the data resource and the access path; The updated second accounting information table and the structured query language SQL statement corresponding to the second accounting information table are sent to the blockchain for storage.

4. The data lineage management method according to claim 3, characterized in that: Also includes: The local publishing table is published according to the access path.

5. The data lineage management method according to claim 4, characterized in that: The updating of the second accounting information table according to the local publishing table, the version information of the data resource and the access path includes: Obtaining the publishing time of the local publishing table; The table name of the local publishing table, the version information of the data resource, the access path and the publishing time are written into the second accounting information table; wherein, If the local publishing table is created locally, NA is written into the second accounting information table as the table name of the local publishing table.

6. The data lineage management method according to claim 1, characterized in that: in, There are multiple local calculation tables, and according to the fusion calculation strategy, the publishing table and the local calculation table are fused and calculated to generate a new publishing table, including: According to the fusion calculation strategy, the plurality of local calculation tables are sequentially fused with the publishing table to generate the new publishing table.

7. The data lineage management method according to claim 1, characterized in that: Also includes: The new publishing table is published according to the new access path.

8. The data lineage management method according to claim 1, characterized in that: The updating of the first accounting information table according to the new publishing table, the version information and the new access path includes: Obtain the new release time of the new release table; The table name of the newly published table, the version information, the new access path and the new publishing time are written into the first accounting information table.

9. A data lineage management method, characterized in that: include: The data management node obtains the updated second accounting information table and the structured query language SQL statement as described in claim 3 from the blockchain; Data lineage analysis is performed based on the updated second accounting information table and the structured query language SQL statement to generate a data lineage analysis result.

10. The data lineage management method according to claim 9, characterized in that: Performing data lineage analysis according to the updated second accounting information table and the structured query language SQL statement to generate a data lineage analysis result, including: Iteratively analyzing each row of data in the updated second accounting information table according to the structured query language SQL statement to obtain the bloodline path information corresponding to each row of data; Data lineage analysis is performed on the lineage path information corresponding to each row of data to generate a data lineage analysis result.

11. A data lineage management device, characterized in that: include: A first acquisition module is used for a data node to acquire an external access path, and acquire an external publishing table published by other data nodes according to the external access path, or receive a data resource publishing instruction, wherein the data resource publishing instruction includes a publishing table of a data resource, and uses the publishing table of the data resource as a local publishing table, and uses the local publishing table or the external publishing table as a publishing table; A second acquisition module, configured for the data node to acquire a local calculation table stored locally after acquiring the publishing table; A fusion calculation module, used for performing fusion calculation on the release table and the calculation table according to a fusion calculation strategy to generate a new release table, wherein the new release table includes version information; A third acquisition module, configured to register with a data-oriented architecture DOA registration center according to the new publication table to acquire a new access path from the DOA registration center; A determination module, configured to determine a first accounting information table corresponding to the new access path according to the new access path; An updating module, configured to update the first accounting information table according to the new publishing table, the version information and the new access path; A sending module is used to send the updated first accounting information table and the structured query language SQL statement corresponding to the first accounting information table to the blockchain for storage, so that the data management node obtains the structured query language SQL statement from the blockchain.

12. A data lineage management device, characterized in that: include: An acquisition module, used for the data management node to acquire the updated second accounting information table and the structured query language SQL statement as described in claim 3 from the blockchain; The analysis module is used to perform data lineage analysis based on the updated second accounting information table and the structured query language SQL statement to generate a data lineage analysis result.

13. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for data lineage management as claimed in any one of claims 1 to 10 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data lineage management method as described in any one of claims 1 to 10 is implemented.

Images