Data lineage construction method, device, equipment and medium

Abstract

The present invention relates to a method, device, equipment and medium for constructing data lineage. The method comprises: a data processing step, dividing metadata into configuration state metadata and execution state metadata; a data construction step, processing the configuration state metadata through a big data platform to form a lineage result table, and importing it into a graph database; a data update step, triggering a real-time update of the graph database through a Kafka message notification when the configuration state metadata is changed, and updating the graph database after the execution state data is processed through the big data platform. The present invention constructs data lineage using an offline and real-time update method, and uses a graph database to save lineage data, so that the timeliness and visibility of lineage updates are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data lineage construction method, device, equipment and medium. Background Art

[0002] Data lineage, also known as data lineage, data origin, and data genealogy, refers to a naturally formed relationship between data, similar to human bloodline, during the entire life cycle of data, from data generation, processing, processing, fusion, circulation to final extinction. In simple terms, it is the upstream and downstream source and destination relationship between data - where the data comes from and where it goes. Data lineage not only involves the physical flow of data, but also includes the logical relationship and conversion process of data.

[0003] Products, operations, development, modelers, and strategists all need to query the upstream and downstream relationships of various types of business objects, but there is currently no system page that can provide the upstream and downstream lineage query function. Usually, the query can only be performed by running SQL through the data warehouse table. This method has the following problems: 1. Manually collect large data tables, and the data is incomplete; 2. Manually write SQL, which is troublesome and the accuracy cannot be guaranteed; 3. It is time-consuming and the communication cost is high. It can be seen that the existing method cannot intuitively express the upstream and downstream relationships of business objects, does not integrate data assets, and has no way to obtain data lineage in real time.

[0004] To address the above problems, no effective solution has been proposed yet. Summary of the invention

[0005] Based on this, it is necessary to provide a data lineage construction method, device, equipment and medium for the above technical problems. The present invention uses offline and real-time update methods to construct data lineage, and uses a graph database to save lineage data, so that the timeliness and visibility of lineage updates are improved.

[0006] According to a first aspect of the present invention, there is provided a data lineage construction method, comprising:

[0007] Data processing step, dividing metadata into configuration state metadata and execution state metadata;

[0008] Data construction step, processing the configuration state metadata through a big data platform to form a blood relationship result table, and importing it into a graph database;

[0009] In the data update step, changes in configuration metadata are triggered through Kafka message notifications to update the graph database in real time, and execution data is processed through the big data platform to update the graph database.

[0010] In some embodiments, it also includes a query step, in which a user reads the graph database through a front-end application to perform information query.

[0011] In some embodiments, the information query includes at least one of a bloodline information query, a call information query, and a bloodline quality report query.

[0012] In some embodiments, the big data platform is the Dataphin big data platform.

[0013] In some embodiments, the replica data synchronization of the Kafka selects a strong consistency strategy and a weak consistency strategy according to the number of accesses within the partition time period t;

[0014] In the strong consistency strategy, the leader replica waits for all ISR replicas to confirm receipt of the data and then returns a confirmation message;

[0015] The weak consistency strategy allows the update of the follower copy to lag behind the leader copy.

[0016] In some embodiments, selecting a strong consistency strategy and a weak consistency strategy according to the number of accesses within the partition time period t includes:

[0017] The consumption heat is calculated based on the number of visits within the partition time period t. The calculation formula is as follows:

[0018] H i =∈(E i / T i )+(1-∈)H (i-1)

[0019] F i =W / T i

[0020] Among them, Hi represents the consumption popularity of the partition in the i-th time period Ti, and H i-1 Indicates the i-1th time period T i-1 Consumption heat of inner partition, E i / T i is the access popularity of the message in period Ti, Ei is the number of accesses to the partition in period Ti, W is the offset of the partition in Ti, and Fi is the update frequency of the follower copy in period Ti;

[0021] When the consumption heat Hi is greater than the update frequency Fi, a strong consistency strategy is selected.

[0022] When the consumption heat Hi is not greater than the update frequency Fi, a weak consistency strategy is selected.

[0023] According to a second aspect of the present invention, there is provided a data lineage construction device, comprising:

[0024] A data processing module, used for dividing metadata into configuration state metadata and execution state metadata;

[0025] A data construction module is used to process the configuration state metadata through a big data platform to form a blood relationship result table, and import it into a graph database;

[0026] The data update module is used to update the graph database by triggering real-time updates of configuration-state metadata changes through Kafka message notifications, and to update the graph database after data processing of execution-state data through the big data platform.

[0027] According to a third aspect of the present invention, there is provided a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0028] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method of any of the above embodiments are implemented.

[0029] The beneficial effects achieved by the present invention are as follows:

[0030] 1. The present invention constructs data lineage by using offline and real-time update methods, processes and integrates data assets, and makes the lineage update time reach quasi-real time. By using a graph database to save lineage data, the lineage relationship of business objects is more intuitively displayed, providing a unified entry for lineage query.

[0031] 2. Based on the number of accesses to the Kafka partition, the consumption popularity and update frequency are calculated. By comparing the two, the synchronization method for the Kafka replica data is selected, which reduces the cluster load while ensuring data reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 are flow charts of some embodiments of the data lineage construction method of the present invention;

[0033] Figure 2 are flow charts of other embodiments of the data lineage construction method of the present invention;

[0034] Figure 3 It is a query diagram of the strategic kinship service of the present invention;

[0035] Figure 4 It is a schematic diagram of the structure of some embodiments of the data lineage construction device of the present invention;

[0036] Figure 5 It is a diagram of the internal structure of a computer device used to implement some embodiments of the present invention. DETAILED DESCRIPTION

[0037] Embodiments of the present invention will be described more fully below with reference to the accompanying drawings, in which embodiments of the present invention are shown. However, the present invention can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein.

[0038] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an", "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that when used herein, the term "comprising" specifies the presence of the claimed features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof.

[0039] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meanings as commonly understood by those of ordinary skill in the art to which the present disclosure belongs. The terms used herein should be interpreted as having the meanings consistent with their meanings in the context of this specification and in the relevant field, and should not be interpreted in an idealized or overly formal sense, unless specifically defined as such herein.

[0040] Configuration metadata: used to build data lineage relationships, reflecting information such as dependencies and flows between data.

[0041] Execution metadata: information used to calculate data call volume and track actual data usage and access.

[0042] Kafka: An open source distributed message queue system.

[0043] Graph database: Graph database (GDB) is a data management system that uses nodes and edges as basic storage units and is designed with the principle of efficient storage and query of graph data.

[0044] Leader replica: The primary replica of a Kafka partition, responsible for handling all read and write operations related to that partition.

[0045] Follower replica: A backup copy of a Kafka partition, responsible for reading data from the Leader replica and synchronizing it.

[0046] Figure 1 Flowcharts showing some embodiments of the data lineage construction method of the present invention.

[0047] like Figure 1 As shown, the method includes:

[0048] S102 is a data processing step, in which the metadata is divided into configuration state metadata and execution state metadata;

[0049] S104 is a data construction step, in which the configuration state metadata is processed through a big data platform to form a blood relationship result table, and then imported into a graph database; wherein the big data platform is the Dataphin big data platform.

[0050] In S106, data update step, changes in configuration state metadata are triggered through Kafka message notification to update the graph database in real time, and execution state data is processed through the big data platform to update the graph database.

[0051] Kafka is an open source distributed message queue system with high throughput and high fault tolerance. However, in actual use, it is difficult for existing synchronization methods to strike a balance between resource overhead and data reliability. Currently, there are two mainstream synchronization methods for Kafka. One is strong consistency, which synchronizes all replicas so that all replicas are in a completely consistent state at a certain moment, ensuring data consistency between each replica, avoiding the risk of data loss and inconsistency, and having higher reliability, but at the same time, the performance overhead is large. The other is weak consistency, which means that different replicas of a message may be in an inconsistent state at a certain moment, but will eventually reach consistency. In some cases, consumers may read old data that some replicas have not yet synchronized, but over time, all replicas will eventually synchronize and remain consistent. Weak consistency has better performance, but replicas may lose data at certain moments.

[0052] Therefore, in order to reduce resource overhead while taking into account data reliability, the present invention proposes a new synchronization strategy. Kafka replica data synchronization selects strong consistency strategy and weak consistency strategy according to the number of accesses within the partition time period t; in the strong consistency strategy, the leader replica waits for all ISR replicas to confirm that they have received the data and then returns a confirmation message; in the weak consistency strategy, the follower replica is allowed to lag behind the leader replica in terms of update. The strong consistency strategy and weak consistency strategy are selected according to the number of accesses within the partition time period t, including:

[0053] The consumption heat is calculated based on the number of visits within the partition time period t. The calculation formula is as follows:

[0054] H i =∈(E i / T i )+(1-∈)H (i-1)

[0055] F i =W / T i

[0056] Among them, H i represents the consumption popularity of the partition in the i-th time period Ti, H i-1 Indicates the i-1th time period T i-1 Consumption heat of inner partition, E i / T i is the access popularity of the message in period Ti, Ei is the number of accesses to the partition in period Ti, W is the offset of the partition in Ti, and Fi is the update frequency of the follower copy in period Ti; i-1 The introduction of avoids the impact of visit number fluctuations on the accuracy of Hi; the impact of the current visit number and the historical visit number on Hi is adjusted by the value of ∈, making Hi more accurate.

[0057] When the consumption heat Hi is greater than the update frequency Fi, a strong consistency strategy is selected.

[0058] When the consumption heat Hi is not greater than the update frequency Fi, a weak consistency strategy is selected.

[0059] That is, when the consumption rate of the partition is greater than the update rate of the partition follower, the strong consistency strategy is adopted, otherwise the weak consistency strategy is adopted. This synchronization strategy reduces resource overhead while ensuring data reliability during daily use.

[0060] In some embodiments, the method further includes a query step in which a user reads the graph database through a front-end application to query information, wherein the information query includes at least one of a blood relationship information query, a call information query, and a blood relationship quality report query.

[0061] Figure 2 Flowcharts showing other embodiments of the data lineage construction method of the present invention.

[0062] like Figure 2 As shown, Dataphin is used to extract data from the configuration state metadata table, and after data processing, the lineage result table is obtained, which is imported through the graph database import task; configuration state metadata changes are updated in real time through Kafka in the graph database; Dataphin is used to extract data from the execution state data, and after metadata processing and call volume statistics, the graph database update task is executed to update the graph data. Figure 3 The query diagram of the strategic kinship service of the present invention is shown as follows: Figure 3 As shown, users can query the policy lineage service through the front-end application.

[0063] Figure 4 A schematic diagram showing the structures of some embodiments of the data lineage construction device of the present invention.

[0064] like Figure 4 As shown, the data lineage construction device in the embodiment includes:

[0065] The data processing module 100 is used to divide the metadata into configuration state metadata and execution state metadata;

[0066] A data construction module 200 is used to process the configuration state metadata through a big data platform to form a blood relationship result table, and import it into a graph database;

[0067] The data update module 300 is used to update the graph database by triggering real-time updates through Kafka message notifications when changes in configuration state metadata are made, and to update the graph database after data processing of execution state data is performed through the big data platform.

[0068] For the specific definition of a data lineage construction device, please refer to the definition of a data lineage construction method above, which will not be repeated here. Each module in the above-mentioned data lineage construction device can be implemented in whole or in part through software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0069] The present invention also provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above-mentioned data lineage construction method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc. Those skilled in the art can understand that Figure 5The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0070] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data lineage construction method is implemented.

[0071] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0072] So far, the embodiments of the present invention have been described in detail. In order to avoid obscuring the concept of the present invention, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein.

[0073] Although some specific embodiments of the present invention have been described in detail by way of examples, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should be understood by those skilled in the art that the above embodiments may be modified or some technical features may be replaced by equivalents without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A data lineage construction method, characterized in that: include: Data processing step, dividing metadata into configuration state metadata and execution state metadata; Data construction step, processing the configuration state metadata through a big data platform to form a blood relationship result table, and importing it into a graph database; In the data update step, changes in configuration metadata are triggered through Kafka message notifications to update the graph database in real time, and execution data is processed through the big data platform to update the graph database.

2. The data lineage construction method according to claim 1, characterized in that: Also includes, In the query step, the user reads the graph database through the front-end application to perform information query.

3. The data lineage construction method according to claim 2, characterized in that: The information query includes at least one of a blood relationship information query, a call information query, and a blood relationship quality report query.

4. The data lineage construction method according to claim 1, characterized in that: The big data platform is the Dataphin big data platform.

5. The data lineage construction method according to claim 1, characterized in that: The Kafka replica data synchronization selects a strong consistency strategy and a weak consistency strategy according to the number of accesses within the partition time period t; In the strong consistency strategy, the leader replica waits for all ISR replicas to confirm receipt of the data and then returns a confirmation message; The weak consistency strategy allows the update of the follower copy to lag behind the leader copy.

6. The data lineage construction method according to claim 5, characterized in that: The selecting of a strong consistency strategy and a weak consistency strategy according to the number of accesses within the partition time period t includes: The consumption heat is calculated based on the number of visits within the partition time period t. The calculation formula is as follows: H i =∈(E i / T i )+(1-∈)H (i-1) F i =W / T i Among them, Hi represents the consumption popularity of the partition in the i-th time period Ti, and H i-1 Indicates the i-1th time period T i-1 Consumption heat of inner partition, E i / T i is the access popularity of the message in period Ti, Ei is the number of accesses to the partition in period Ti, W is the offset of the partition in Ti, and Fi is the update frequency of the follower copy in period Ti; When the consumption heat Hi is greater than the update frequency Fi, a strong consistency strategy is selected. When the consumption heat Hi is not greater than the update frequency Fi, a weak consistency strategy is selected.

7. A data lineage construction device, characterized in that: include: A data processing module, used for dividing metadata into configuration state metadata and execution state metadata; A data construction module is used to process the configuration state metadata through a big data platform to form a blood relationship result table, and import it into a graph database; The data update module is used to update the graph database by triggering real-time updates of configuration-state metadata changes through Kafka message notifications, and to update the graph database after data processing of execution-state data through the big data platform.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images