Data bloodline circulation link generation method, circulation link generation device and medium

Abstract

This invention discloses a method, apparatus, and medium for generating data lineage flow links. The method includes the following steps: when an application is detected to be running, the target code for producing and consuming messages in the Kafka Toolkit corresponding to the application is determined; when the target code is detected to be executed, data lineage information generated by the application during production and consumption is collected according to the data lineage acquisition program based on the tracking address of the Kafka Toolkit; the data lineage information is associated with the registration information of the Kafka Toolkit, and the data lineage information is distributed to a Kafka data reading unit; all the data lineage information from the Kafka data reading unit is reported to a data lineage analysis platform to obtain the flow link of the data lineage information. By embedding a data lineage acquisition program into the Kafka Toolkit, a visualized flow link of data lineage information is obtained.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus and medium for generating data lineage transfer links. Background Technology

[0002] Data lineage refers to a relationship similar to blood ties in human society that forms between data during the process of data generation, processing, flow, and disappearance.

[0003] In conventional data lineage collection methods, open-source systems such as Apache Atlas (a core capability for metadata governance) and Netflix Metacat (a metadata service) are typically used to collect metadata from data sources to build a data asset catalog. This catalog then categorizes and governs data assets from the data source's perspective, maintaining the lineage relationships between data sources. However, current data lineage collection methods only capture the lineage relationships between data sources from the data source's perspective, lacking application-level data lineage management. This means it's impossible to perceive which specific application produced, processed, or consumed the data, and the flow of data between applications is invisible. The data flow chain cannot be fully presented, making it difficult to accurately pinpoint the specific application when data lineage anomalies occur, thus hindering maintenance.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, and medium for generating data lineage transfer links, thereby solving the problem in the prior art where abnormalities in data lineage cannot be accurately located to specific applications, leading to difficulties in maintenance.

[0006] To achieve the above objectives, the present invention provides a method for generating data lineage transfer links, the method comprising the following steps:

[0007] When an application is detected to be running, the target code for producing and consuming messages in the Kafka toolkit corresponding to the application is determined.

[0008] When the execution of the target code is detected, the data lineage information generated by the application during production and consumption is collected according to the data lineage collection program of the Kafka toolkit's tracking address.

[0009] The data lineage information is associated with the registration information of the Kafka toolkit, and the data lineage information is distributed to the Kafka data reading unit.

[0010] All the data lineage information of the Kafka data reading unit is reported to the data lineage analysis platform to obtain the flow link of the data lineage information.

[0011] Optionally, after the step of determining the target code for producing and consuming messages in the Kafka toolkit corresponding to the application when the application is detected to have entered the running state, the method further includes:

[0012] Determine the code logic location of the target code, and add data interception instructions at the code logic location;

[0013] The step of collecting data lineage information generated during the production and consumption of the application based on the data lineage collection program of the Kafka toolkit's tracking address when the target code is detected to be executed includes:

[0014] When the data interception command is triggered, it is determined that the target code has been executed;

[0015] Based on the data lineage collection program at the aforementioned embedded address, data lineage information generated by the application during production and consumption is collected.

[0016] Optionally, before the step of collecting data lineage information generated during the production and consumption of the application based on the data lineage collection program of the Kafka toolkit's tracking address when the target code is detected to be executed, the method further includes:

[0017] Obtain the first storage location where the Kafka toolkit performs production and consumption actions;

[0018] The first storage location is used as the embedding address, and the data lineage collection program is embedded in the embedding address.

[0019] Optionally, before the step of collecting data lineage information generated during the production and consumption of the application based on the data lineage collection program of the Kafka toolkit's tracking address when the target code is detected to be executed, the method further includes:

[0020] Obtain the second storage location for initializing the production client and the consumer client in the Kafka toolkit;

[0021] The second storage location is used as the embedding address, and the data lineage collection program is embedded in the embedding address.

[0022] Optionally, before the step of determining the target code for producing and consuming messages in the Kafka toolkit corresponding to the application when the application is detected to have entered the running state, the method further includes:

[0023] Add the startup parameters of the data lineage collection program to the startup command line of the application;

[0024] When the application is detected to be running, the data lineage collection program is started based on the startup parameters.

[0025] Optionally, before the step of associating the data lineage information with the registration information of the Kafka toolkit and distributing the data lineage information to the Kafka data reading unit, the method further includes:

[0026] The registration information of the Kafka toolkit is obtained based on the JMX monitoring module of the application.

[0027] The step of associating the data lineage information with the registration information of the Kafka toolkit and distributing the data lineage information to the Kafka data reading unit includes:

[0028] Associating the first client number of the data lineage information with the second client number of the registration information, and determining the transmission method of the data lineage information;

[0029] When the transmission method is interface transmission, the configuration rules for the interface application, cluster, and application level are determined;

[0030] The data lineage information is distributed to the Kafka data reading unit according to the configuration rules.

[0031] Optionally, after the step of associating the data lineage information with the registration information of the Kafka toolkit and distributing the data lineage information to the Kafka data reading unit, the method further includes:

[0032] Upon receiving an update instruction for the data lineage information, the target lineage data corresponding to the update instruction and the lineage data to be updated on the data lineage analysis platform are determined.

[0033] The target lineage data is used to update and replace the lineage data to be updated.

[0034] Optionally, after the step of reporting all the data lineage information of the Kafka data reading unit to the data lineage analysis platform to obtain the flow path of the data lineage information, the method further includes:

[0035] When a data transfer failure is detected at a node in the transfer link, the application node corresponding to the abnormal node is traced according to the transfer link.

[0036] Obtain the cause of the anomaly in the application node and determine the scope of business impact corresponding to the cause of the anomaly;

[0037] Based on the cause of the anomaly and the scope of its impact on the business, an anomaly alarm message is output.

[0038] In addition, to achieve the above objectives, the present invention also provides a data lineage generation apparatus, which includes a memory, a processor, and a data lineage generation program stored in the memory and executable on the processor. When the data lineage generation program is executed by the processor, it implements the steps of the data lineage generation method described above.

[0039] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a data lineage generation program, wherein when the data lineage generation program is executed by a processor, it implements the steps of the data lineage generation method described above.

[0040] This invention provides a method, apparatus, and medium for generating data lineage flow links. When an application is detected to be running, the target code for producing and consuming messages in the Kafka Toolkit corresponding to the application is determined. Upon monitoring the execution of the target code, a data lineage acquisition program based on the Kafka Toolkit's embedding address collects the data lineage information generated by the application during production and consumption. This data lineage information is then associated with the Kafka Toolkit's registration information. The data lineage information is distributed to a Kafka data reading unit, and all data lineage information from the Kafka data reading unit is reported to a data lineage analysis platform, thus obtaining the flow link of the data lineage information. It can be seen that by associating and binding the collected data lineage information with the Kafka Toolkit's registration information, and by reporting the data lineage information to the data lineage analysis platform based on the Kafka data reading unit, a visualized data lineage flow link is obtained. Based on this, by visualizing the data flow path, it is possible to accurately locate the specific application when an anomaly occurs in the data lineage of a certain application, thereby improving the maintenance efficiency when an anomaly occurs in the data lineage. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the invention. To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0042] Figure 1 This is a flowchart illustrating the first embodiment of the data lineage transfer link generation method of the present invention.

[0043] Figure 2 This is a schematic diagram of the topology of the data lineage flowing between the application and the Kafka toolkit, generated by the data lineage flow link generation method of the present invention.

[0044] Figure 3 This is a flowchart illustrating a second embodiment of the data lineage transfer link generation method of the present invention;

[0045] Figure 4 This is a schematic diagram of the data lineage information collection process of the data lineage transfer link generation method of the present invention;

[0046] Figure 5 These are schematic diagrams of the terminal hardware structure of various embodiments of the data lineage transfer link generation method of the present invention.

[0047] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0049] Data lineage has two sources: manual registration and automated collection. Traditional data lineage registration and maintenance uses documents. Manual registration faces significant workload issues related to lineage registration and changes. While digital online registration can reduce workload to some extent, it still cannot guarantee the accuracy and timeliness of data lineage registration and maintenance, cannot obtain real-time information on operational lineage relationships, and cannot achieve real-time monitoring and alerts for data lineage anomalies. Conventional automated data lineage collection methods typically use open-source systems such as Apache Atlas (metadata governance core capabilities) and Netflix Metacat (metadata service) to collect metadata from data sources to build a data asset catalog. This catalog categorizes and governs data assets from the data source's perspective to maintain lineage relationships between data sources. However, current data lineage collection methods only collect lineage relationships from the data source's perspective, lacking application-level data lineage management. That is, it cannot perceive which specific application produced, processed, or consumed the data, and the flow of data between applications is invisible. The data flow chain cannot be fully presented, making it impossible to accurately locate the specific application when data lineage anomalies occur, thus hindering maintenance.

[0050] To address the aforementioned shortcomings, this invention proposes a method for generating data lineage transfer links, the main solution of which includes the following steps:

[0051] When an application is detected to be running, the target code for producing and consuming messages in the Kafka toolkit corresponding to the application is determined.

[0052] When the execution of the target code is detected, the data lineage information generated by the application during production and consumption is collected according to the data lineage collection program of the Kafka toolkit's tracking address.

[0053] The data lineage information is associated with the registration information of the Kafka toolkit, and the data lineage information is distributed to the Kafka data reading unit.

[0054] All the data lineage information of the Kafka data reading unit is reported to the data lineage analysis platform to obtain the flow link of the data lineage information.

[0055] This invention associates and binds the collected data lineage information with the registration information of the Kafka Toolkit, and reports the data lineage information to a data lineage analysis platform based on the Kafka data reading unit, thereby obtaining a visualized data lineage information flow path. Based on this, by visualizing the flow path, when an anomaly occurs in the data lineage of a certain application, the specific application can be accurately located, thereby improving the maintenance efficiency when data lineage anomalies occur.

[0056] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0057] First Embodiment

[0058] In the first embodiment, before collecting data lineage information from the application, the data lineage collection program needs to be embedded in the application, such as Flink, Spark, or a regular Java application. The address of the embedded point is the address where the Kafka toolkit produces and consumes messages. At the application level, after the data lineage collection program is embedded into the corresponding Kafka toolkit, the application developer intercepts real-time data based on the Kafka toolkit. That is, data lineage can be collected for Kafka Topics (Kafka: an open-source message queue system; Topic: the basic unit of Kafka data read and write operations), and then the data lineage can be aggregated to a data lineage analysis platform to obtain the complete data lineage chain, enabling developers or other users to understand the origin and development of data throughout its entire lifecycle.

[0059] Please refer to Figure 1 In the first embodiment, the method for generating data lineage flow links includes the following steps:

[0060] Step S10: When the application is detected to have entered the running state, determine the target code for producing and consuming messages in the Kafka toolkit corresponding to the application.

[0061] In this embodiment, after the application enters the running state, specific logic code needs to be implanted in the Kafka Toolkit to start the data lineage acquisition program within the application. Therefore, when the application transitions from the closed state to the running state, it is necessary to determine the target code for producing and consuming messages within the Kafka Toolkit corresponding to the application. The code logic location corresponding to the target code is the location where the specific logic code is stored. Based on this, by determining the target code in the Kafka Toolkit and obtaining its code logic address, it is possible to determine whether the target code has been executed based on the specific logic code corresponding to that address.

[0062] Specifically, the specific logic code embedded in the Kafka toolkit can serve as a data interception instruction. That is, when the data lineage acquisition program needs to collect the data lineage information of the application, the data interception instruction at the code logic location of the target code is activated, at which point the data lineage acquisition program needs to be run. Therefore, after determining the target code, it is necessary to determine the code logic location of the target code and add a data interception instruction at that code logic location, thereby ensuring that the data lineage acquisition system can run the data lineage acquisition program based on the interception instruction.

[0063] Optionally, the data lineage acquisition program needs to be woken up and enter standby mode by specific code. That is, the startup parameters of the data lineage acquisition program need to be added to the startup command line of the application, and the data lineage acquisition program is started based on the startup parameters when the application is detected to be running.

[0064] It should be noted that the data interception command is used to run the data lineage acquisition program under certain conditions, thereby collecting the corresponding data lineage information, while the startup parameter adjusts the data lineage acquisition program from the closed state to the standby state.

[0065] Step S20: When the target code is detected to be executed, the data lineage information generated by the application during production and consumption is collected according to the data lineage collection program of the Kafka toolkit's embedding address.

[0066] Specifically, when a data interception instruction is triggered in the code logic location corresponding to the target code, it is determined that the target code has been executed, indicating that data lineage collection is required. At this time, the data lineage collection program based on the tracking addresses of the Kafka toolkit needs to collect the data lineage information generated by the application during production and consumption.

[0067] Optionally, the embedding address of the data lineage acquisition program can be the storage address where the Kafka Toolkit executes production and consumption actions, or it can be the storage address where the Kafka Toolkit initializes the production client and the consumption client. Specifically, before step S20, the following steps are also included: obtaining the first storage location where the Kafka Toolkit executes production and consumption actions, using the first storage location as the embedding address, and embedding the data lineage acquisition program into the embedding address.

[0068] Optionally, before step S20, the following steps may be included: obtaining the second storage location of the initialization production client and consumer client in the Kafka toolkit, using the second storage location as the embedding address, and embedding the data lineage collection program into the embedding address.

[0069] It should be noted that in this embodiment, the data lineage collection program is embedded in the embedding address using Java Agent technology (a bytecode enhancement technology provided by the high-level programming language Java), thereby improving the stability of the data lineage collection program after embedding.

[0070] Step S30: Associate the data lineage information with the registration information of the Kafka toolkit, and distribute the data lineage information to the Kafka data reading unit;

[0071] In this embodiment, in order to ensure that the data acquisition program can obtain the complete data link and thus understand the origin and development of each data lineage, it is necessary to associate the data lineage with the registration information of the Kafka Toolkit, and then distribute the data lineage information to the Kafka data reading unit after association.

[0072] Step S40: Report all the data lineage information of the Kafka data reading unit to the data lineage analysis platform to obtain the flow link of the data lineage information.

[0073] In this embodiment, the Kafka data reading unit stores the data lineage information corresponding to the application's production and consumption of a certain Kafka topic. After obtaining the data lineage information corresponding to the application, the data lineage information needs to be reported to the data lineage analysis platform to obtain the flow link of the data lineage information.

[0074] Optionally, after step S40, the method further includes: when a data transfer failure is detected at a node in the transfer link, tracing the application node corresponding to the abnormal node according to the transfer link, obtaining the cause of the abnormality of the application node, determining the scope of business impact corresponding to the cause of the abnormality, and outputting an abnormal alarm message based on the cause of the abnormality and the scope of business impact. The transfer information corresponding to each node in the transfer link can be obtained through a visual interface. When a node in the link is abnormal, developers can quickly determine the location of the abnormality based on the visual interface and then perform maintenance.

[0075] Specifically, please refer to Figure 2 , Figure 2 This is a topology diagram showing the data flow between applications and the Kafka toolkit. This topology diagram represents the complete flow chain relationship of the data lineage information obtained by the data lineage analysis platform. Flink-A and Spark-Streaming-A are applications. After the data lineage collection program is embedded into the corresponding Kafka toolkits of Flink-A and Spark-Streaming-A, the link information when the application consumes data from Kafka Topics A, B, and C, and when it produces data from Kafka Topics D and E, is collected. Subsequently, after the Flink-F application consumes Kafka Topics D and E, the Kafka Topics D and E produced by applications Flink-A and Spark-Streaming-A are sent to Kafka Topic G. For an example, please refer to... Figure 2 When the Kafka Topic D node experiences an anomaly, Flink-F cannot receive messages from Kafka Topic D. At this time, the abnormal state of the node can be accurately located, and maintenance and handling can be carried out as soon as possible.

[0076] Therefore, once a complete data lineage diagram is obtained, when an anomaly occurs in a node of the lineage, the diagram can be used to trace the anomaly and send alarm information to the responsible parties of the applications and data assets in the lineage. This allows for quick location of the cause of the anomaly, determination of the scope of business affected by the anomaly, and provision of corresponding SLA guarantees based on the SLA (Service Level Agreement) service level of the applications in the lineage, thereby minimizing losses.

[0077] Optionally, when it is necessary to update the data lineage information corresponding to a certain production or consumption node in the application, the data lineage information corresponding to that node can be updated and replaced. That is, when receiving the data lineage information update instruction, the target lineage data corresponding to the update instruction and the lineage data to be updated in the data lineage analysis platform are determined, and the lineage data to be updated is updated and replaced based on the target lineage data.

[0078] In the technical solution disclosed in this embodiment, when the application enters the running state and the target code for producing and consuming messages in the Kafka Toolkit is executed, data lineage is collected for the Kafka Toolkit. After the data lineage information is collected, it is reported to the data lineage analysis platform to obtain a complete and visualized data lineage chain. This helps developers understand the ins and outs of data throughout the entire lineage chain lifecycle. It also enables rapid location of the cause of the anomaly when a node in the lineage chain is abnormal, determination of the scope of business affected by the anomaly, and rapid maintenance, thereby improving the maintenance efficiency of data lineage information.

[0079] Second Embodiment

[0080] Please refer to Figure 3 In the second embodiment, based on this first embodiment, step S30 includes:

[0081] Step S31: Associate the first client number of the data lineage information with the second client number of the registration information, and determine the transmission method of the data lineage information;

[0082] In this embodiment, the registration information of the Kafka toolkit can be obtained according to the JMX (Java Management Extensions) monitoring module of the application. The registration information monitored by the JMX monitoring module includes the Kafka Producer, Kafka Consumer, the consumed Topic (a type of message), and the corresponding client ID. Then, the client ID information collected by the JMX monitoring module is associated with the corresponding client ID information in the data lineage information to obtain the link information between the application and a specific Topic of the Kafka toolkit's real-time data source. Finally, the transmission method for the data lineage information is determined, and the data lineage information is transmitted based on this transmission method.

[0083] Step S32: When the transmission mode is interface transmission, determine the configuration rules for the interface application, cluster, and application level.

[0084] In this embodiment, after determining the configuration rules for the interface application, cluster, and its own application level, it can be ensured that the current data lineage information can be transmitted to the Kafka data reading unit.

[0085] Step S33: Distribute the data lineage information to the Kafka data reading unit according to the configuration rules.

[0086] Specifically, please refer to Figure 4 In this embodiment, when the data acquisition program acquires data lineage information (i.e., Figure 4 After obtaining the metadata information shown, the client ID of the Kafka tool registration information collected by the JMX monitoring module is associated and bound. Then, the corresponding transmission rule is selected according to the current lineage data transmission method, and the data is distributed to the Kafka data reading unit (i.e., Kafka topic). This ensures that the Kafka data reading unit reports all data lineage information to the data analysis platform to obtain the complete link information of the data lineage information.

[0087] In the technical solution disclosed in this embodiment, after collecting data lineage information, the client ID of the data lineage information is bound to the client ID corresponding to the registration information of the Kafka toolkit monitored by JMX, thereby obtaining the link information between the application and the real-time data source of the Kafka toolkit. Based on this, when an anomaly occurs in a node of the data lineage information, the abnormal location can be quickly located, thereby improving maintenance efficiency.

[0088] Reference Figure 5 , Figure 5 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention.

[0089] like Figure 5 As shown, the terminal may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a network interface 1003, and a memory 1004. The communication bus 1002 is used to enable communication between these components. The network interface 1003 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1004 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. Alternatively, the memory 1004 may be a storage device independent of the aforementioned processor 1001.

[0090] Those skilled in the art will understand that Figure 5The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0091] like Figure 5 As shown, the memory 1004, which serves as a computer storage medium, may include an operating system, a data storage module, a network communication module, and a data transfer generation program.

[0092] exist Figure 5 In the terminal shown, network interface 1003 is mainly used to connect to the backend server and communicate with it; processor 1001 can call the data transfer generation program stored in memory 1004 and perform the following operations:

[0093] When an application is detected to be running, the target code for producing and consuming messages in the Kafka toolkit corresponding to the application is determined.

[0094] When the execution of the target code is detected, the data lineage information generated by the application during production and consumption is collected according to the data lineage collection program of the Kafka toolkit's tracking address.

[0095] The data lineage information is associated with the registration information of the Kafka toolkit, and the data lineage information is distributed to the Kafka data reading unit.

[0096] All the data lineage information of the Kafka data reading unit is reported to the data lineage analysis platform to obtain the flow link of the data lineage information.

[0097] Furthermore, the processor 1001 can call the flow link generation program stored in the memory 1004 and also perform the following operations:

[0098] Determine the code logic location of the target code, and add data interception instructions at the code logic location;

[0099] The step of collecting data lineage information generated during the production and consumption of the application based on the data lineage collection program of the Kafka toolkit's tracking address when the target code is detected to be executed includes:

[0100] When the data interception command is triggered, it is determined that the target code has been executed;

[0101] Based on the data lineage collection program at the aforementioned embedded address, data lineage information generated by the application during production and consumption is collected.

[0102] Furthermore, the processor 1001 can call the flow link generation program stored in the memory 1004 and also perform the following operations:

[0103] Obtain the first storage location where the Kafka toolkit performs production and consumption actions;

[0104] The first storage location is used as the embedding address, and the data lineage collection program is embedded in the embedding address.

[0105] Furthermore, the processor 1001 can call the flow link generation program stored in the memory 1004 and also perform the following operations:

[0106] Obtain the second storage location for initializing the production client and the consumer client in the Kafka toolkit;

[0107] The second storage location is used as the embedding address, and the data lineage collection program is embedded in the embedding address.

[0108] Furthermore, the processor 1001 can call the flow link generation program stored in the memory 1004 and also perform the following operations:

[0109] Add the startup parameters of the data lineage collection program to the startup command line of the application;

[0110] When the application is detected to be running, the data lineage collection program is started based on the startup parameters.

[0111] Furthermore, the processor 1001 can call the flow link generation program stored in the memory 1004 and also perform the following operations:

[0112] The registration information of the Kafka toolkit is obtained based on the JMX monitoring module of the application.

[0113] The step of associating the data lineage information with the registration information of the Kafka toolkit and distributing the data lineage information to the Kafka data reading unit includes:

[0114] Associating the first client number of the data lineage information with the second client number of the registration information, and determining the transmission method of the data lineage information;

[0115] When the transmission method is interface transmission, the configuration rules for the interface application, cluster, and application level are determined;

[0116] The data lineage information is distributed to the Kafka data reading unit according to the configuration rules.

[0117] Furthermore, the processor 1001 can call the flow link generation program stored in the memory 1004 and also perform the following operations:

[0118] Upon receiving an update instruction for the data lineage information, the target lineage data corresponding to the update instruction and the lineage data to be updated on the data lineage analysis platform are determined.

[0119] The target lineage data is used to update and replace the lineage data to be updated.

[0120] Furthermore, the processor 1001 can call the flow link generation program stored in the memory 1004 and also perform the following operations:

[0121] When a data transfer failure is detected at a node in the transfer link, the application node corresponding to the abnormal node is traced according to the transfer link.

[0122] Obtain the cause of the anomaly in the application node and determine the scope of business impact corresponding to the cause of the anomaly;

[0123] Based on the cause of the anomaly and the scope of its impact on the business, an anomaly alarm message is output.

[0124] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the control terminal to implement the process steps of the embodiments of the above methods.

[0125] Therefore, the present invention also provides a computer-readable storage medium storing a data lineage transfer link generation program, which, when executed by a processor, implements the various steps of the data lineage transfer link generation method as described in the above embodiments.

[0126] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0127] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0132] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0133] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0134] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for generating data lineage transfer links, characterized in that, The method for generating the data lineage flow link includes: When an application is detected to be running, the target code for producing and consuming messages in the Kafka toolkit corresponding to the application is determined. Determine the code logic location of the target code, and add data interception instructions at the code logic location; When the data interception command is triggered, it is determined that the target code has been executed; When the execution of the target code is detected, the data lineage information generated by the application during production and consumption is collected according to the data lineage collection program of the Kafka toolkit's tracking address. The registration information of the Kafka toolkit is obtained based on the JMX monitoring module of the application. Associating the data lineage information with the registration information of the Kafka Toolkit, and distributing the data lineage information to the Kafka data reading unit, includes: associating the first client number of the data lineage information with the second client number of the registration information, and determining the transmission method of the data lineage information; when the transmission method is interface transmission, determining the configuration rules of the interface application, cluster, and application level; and distributing the data lineage information to the Kafka data reading unit according to the configuration rules. All the data lineage information of the Kafka data reading unit is reported to the data lineage analysis platform to obtain the flow link of the data lineage information.

2. The method for generating data lineage transfer links as described in claim 1, characterized in that, Before the step of collecting data lineage information generated during the production and consumption of the application based on the data lineage collection program of the Kafka toolkit's tracking address when the target code is detected to be executed, the method further includes: Obtain the first storage location where the Kafka toolkit performs production and consumption actions; The first storage location is used as the embedding address, and the data lineage collection program is embedded in the embedding address.

3. The method for generating data lineage transfer links as described in claim 1, characterized in that, Before the step of collecting data lineage information generated during the production and consumption of the application based on the data lineage collection program of the Kafka toolkit's tracking address when the target code is detected to be executed, the method further includes: Obtain the second storage location for initializing the production client and the consumer client in the Kafka toolkit; The second storage location is used as the embedding address, and the data lineage collection program is embedded in the embedding address.

4. The method for generating data lineage transfer links as described in claim 1, characterized in that, Before the step of determining the target code for producing and consuming messages in the Kafka toolkit corresponding to the application when the application is detected to have entered the running state, the method further includes: Add the startup parameters of the data lineage collection program to the startup command line of the application; When the application is detected to be running, the data lineage collection program is started based on the startup parameters.

5. The method for generating data lineage transfer links as described in claim 1, characterized in that, After the step of associating the data lineage information with the registration information of the Kafka toolkit and distributing the data lineage information to the Kafka data reading unit, the method further includes: Upon receiving an update instruction for the data lineage information, the target lineage data corresponding to the update instruction and the lineage data to be updated on the data lineage analysis platform are determined. The target lineage data is used to update and replace the lineage data to be updated.

6. The method for generating data lineage transfer links as described in claim 1, characterized in that, After the step of reporting all the data lineage information of the Kafka data reading unit to the data lineage analysis platform to obtain the flow path of the data lineage information, the method further includes: When a data transfer failure is detected at a node in the transfer link, the application node corresponding to the abnormal node is traced according to the transfer link. Obtain the cause of the anomaly in the application node and determine the scope of business impact corresponding to the cause of the anomaly; Based on the cause of the anomaly and the scope of its impact on the business, an anomaly alarm message is output.

7. A transfer link generation device, characterized in that, The apparatus includes: a memory, a processor, and a data link generation program stored in the memory and executable on the processor, wherein when executed by the processor, the data link generation program implements the steps of the data lineage data link generation method as claimed in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data lineage generation program, which, when executed by a processor, implements the steps of the data lineage generation method as described in any one of claims 1 to 6.

Images