A data bloodline analysis method, device, medium, electronic equipment and product

Abstract

The application discloses a data bloodline analysis method and device, a medium, an electronic equipment and a product. The method comprises the following steps: acquiring an end operation node of a target scene, determining an operation link of the target scene based on the end operation node; extracting operation information of a plurality of operation nodes in the operation link, wherein the operation information comprises execution host information and operation time information; screening historical execution statements based on the execution host information and the operation time information of each operation node to obtain target execution statements; and constructing a data bloodline analysis result of the target scene based on the target execution statements. The data bloodline analysis of the target scene is realized, and the accuracy of the data bloodline analysis result of the target scene is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data lineage analysis method, apparatus, medium, electronic device, and product. Background Technology

[0002] With the development and application of big data technology, the need for data lineage analysis has emerged in the field of big data governance.

[0003] Data lineage analysis is obtained by determining the upstream and downstream relationships between data during data processing. In the operation of a system or organization, different application scenarios exist. However, current data lineage analysis is based on the large amount of data processed during the operation of a system or organization (e.g., including one or more systems), making it impossible to specifically determine the data lineage analysis results for a particular scenario. Summary of the Invention

[0004] This invention provides a data lineage analysis method, apparatus, medium, electronic device, and product, which achieves high accuracy in obtaining data lineage analysis results for a target scenario and is unaffected by data from non-target scenarios.

[0005] According to one aspect of the present invention, a data lineage analysis method is provided, comprising:

[0006] Obtain the end-point operation nodes of the target scenario, and determine the operation chain of the target scenario based on the end-point operation nodes;

[0007] Extract job information from multiple job nodes in the job chain, the job information including execution host information and job time information;

[0008] Based on the execution host information and job time information of each job node, the historical execution statements are filtered to obtain the target execution statement;

[0009] The data lineage analysis results for the target scenario are constructed based on the target execution statement.

[0010] Optionally, the target scenario includes a regulatory scenario, and the end-user node in the regulatory scenario is a node that sends regulatory data to the regulatory equipment.

[0011] Optionally, determining the job chain of the target scenario based on the end job node includes:

[0012] Based on the dependencies between job nodes, the multi-level upstream job nodes of the end job node are determined sequentially, and the end job node and the multi-level upstream job nodes form a job link.

[0013] Optionally, the job time information includes the job start time and job end time;

[0014] The step of filtering historical execution statements based on the execution host information and job time information of each job node to obtain target execution statements includes: determining historical execution statements that match the execution host information of the job node and whose execution timestamps are located between the job start time and the job end time, as target execution statements.

[0015] Optionally, the target execution statement includes source data and target data;

[0016] The step of constructing the data lineage analysis result of the target scenario based on the target execution statement includes: for each target execution statement, extracting the source data and target data in the target execution statement, and establishing the data lineage relationship between the source data and the target data; and forming a data lineage analysis result based on the data lineage relationship corresponding to multiple target execution statements.

[0017] Optionally, the source data and the target data each include one or more of the following: system data, table data, and field data; correspondingly, the data lineage includes system-level lineage, table-level lineage, and field-level lineage.

[0018] According to one aspect of the present invention, a data lineage analysis device is provided, comprising:

[0019] The job link determination module is used to obtain the end job nodes of the target scenario and determine the job link of the target scenario based on the end job nodes;

[0020] The job information extraction module is used to extract job information from multiple job nodes in the job chain. The job information includes execution host information and job time information.

[0021] The statement filtering module is used to filter historical execution statements based on the execution host information and job time information of each job node to obtain the target execution statement;

[0022] The data lineage construction module is used to construct the data lineage analysis results of the target scenario based on the target execution statement.

[0023] Optionally, the job link determination module is used to: determine the multi-level upstream job nodes of the end job node in sequence based on the dependency relationship between job nodes, and the end job node and the multi-level upstream job nodes form a job link.

[0024] Optionally, the job time information includes the job start time and job end time;

[0025] The statement filtering module is used to: determine historical execution statements that match the execution host information of the job node and whose execution timestamps are located between the job start time and the job end time, and use them as target execution statements.

[0026] Optionally, the target execution statement includes source data and target data;

[0027] The data lineage construction module is used to extract source data and target data from each target execution statement, establish the data lineage relationship between the source data and target data, and form a data lineage analysis result based on the data lineage relationships corresponding to multiple target execution statements.

[0028] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0029] At least one processor; and

[0030] A memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data lineage analysis method according to any embodiment of the present invention.

[0032] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the data lineage analysis method according to any embodiment of the present invention.

[0033] According to another aspect of the present invention, a computer program product is provided, characterized in that the computer program product includes a computer program that, when executed by a processor, implements the data lineage analysis method described in any embodiment of the present invention.

[0034] The technical solution of this invention reduces interfering execution statements by determining the job chain of a target scenario, matching the target execution statement of the job chain, and eliminating historical execution statements outside the target scenario. The data lineage analysis results for the target scenario are constructed using the target execution statements, thus achieving data lineage analysis for the target scenario and improving the accuracy of the data lineage analysis results.

[0035] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a data lineage analysis method provided in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of a job chain provided in an embodiment of the present invention;

[0039] Figure 3 This is a flowchart of a data lineage analysis method provided in an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of a data lineage analysis device provided in an embodiment of the present invention;

[0041] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0043] It should be noted that the terms "first feature data," "second feature data," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] The acquisition, storage, and / or processing of data in the technical solutions involved in this application comply with the relevant provisions of national laws and regulations.

[0045] Example 1

[0046] Figure 1 This is a flowchart of a data lineage analysis method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where suitable and interesting positions are recommended to newly hired employees. The method can be executed by a data lineage analysis device, which can be implemented in hardware and / or software. This data lineage analysis device can be configured in the electronic device provided in this embodiment of the present invention. Figure 1 As shown, the method includes:

[0047] S110. Obtain the end-of-line operation node of the target scenario, and determine the operation link of the target scenario based on the end-of-line operation node.

[0048] S120. Extract job information from multiple job nodes in the job chain, the job information including execution host information and job time information.

[0049] S130. Based on the execution host information and job time information of each job node, the historical execution statements are filtered to obtain the target execution statement.

[0050] S140. Construct the data lineage analysis results of the target scenario based on the target execution statement.

[0051] During the operation of a business system, multiple application scenarios may be included. Different business systems or those belonging to different institutions may include different application scenarios. Taking a financial institution as an example, the business scenarios during the operation of its business system include, but are not limited to, transaction scenarios and regulatory scenarios. The target scenario is any of the above application scenarios and can be determined based on the analysis requirements of data lineage. This target scenario can be pre-set.

[0052] Each application scenario requires execution through a job chain, which includes multiple job nodes. The combination of job nodes in a job chain is variable, and job nodes in different job chains may overlap. It is understandable that there are dependencies between adjacent job nodes in a job chain. By determining the end job node of the application scenario, the other job nodes in the job chain are deduced in reverse to form the job chain.

[0053] Different application scenarios yield different result data. Each result data can correspond to a specific application scenario, and the processing procedure for obtaining that result data can be used to determine the execution process of that application scenario. Taking a regulatory scenario as an example, the result data is regulatory data, which can be in report form (this is not limited here). Each piece of regulatory data corresponds to a regulatory scenario. Optionally, upon detecting regulatory data, a data lineage analysis for the regulatory scenario can be triggered. Similarly, for other application scenarios, a data lineage analysis for the application scenario can be triggered upon detecting the result data of that application scenario.

[0054] The end-point job node in the target scenario is the job node that outputs the result data. Taking the regulatory scenario as an example, the end-point job node in the regulatory scenario is the job node that sends regulatory data to the regulatory equipment. Optionally, the job node can be an ETL (Extract, Transform and Load) job node.

[0055] The task chain for the target scenario is determined based on the end-point task nodes. Specifically, based on the dependencies between task nodes, multi-level upstream task nodes are sequentially determined. These dependencies are pre-configured and can be determined by reading a dependency configuration file. For each upstream task node determined by these dependencies, the upstream task node that the end-point task node depends on is then identified, and so on, until any upstream task node has no further upstream task nodes.

[0056] The end-point operation node and the multi-level upstream operation node form an operation link. Specifically, the end-point operation node and multiple operation nodes in the multi-level upstream operation node can be sorted based on dependency or upstream-downstream relationship, or the operation nodes can be connected in series based on dependency or upstream-downstream relationship.

[0057] For example, see Figure 2 , Figure 2 This is a schematic diagram of a job chain provided by an embodiment of the present invention. The end job nodes of the target scenario can be job node 4 and job node 8. The multi-level upstream job nodes that job node 4 depends on and the multi-level upstream job nodes that job node 8 depends on are determined respectively, so as to obtain the job chains corresponding to job node 4 and job node 8 respectively.

[0058] By identifying the data processed by job nodes in the job chain as data from the target scenario, data lineage analysis is performed using data from the target scenario to avoid interference from data outside the target scenario and improve the accuracy of data lineage analysis. During the job process of each job node, historical execution statements are generated. These historical execution statements can serve as log information, facilitating the querying of the job node's process. These historical execution statements can be SQL (Structured Query Language) statements.

[0059] Historical execution statements can serve as log information, recording relevant data processing information during the job node's operation. By identifying the target execution statement corresponding to the target scenario from all historical execution statements in the business system or organization, interference from other historical execution statements on data lineage analysis can be reduced.

[0060] Each job node can be configured with job information, including but not limited to the job node name, job triggering conditions, condition name, job invocation method, job status, job time information, and execution entity information. The job time information includes the job start time and job end time. After determining the job chain, the job chain can be verified based on the job time of each job node in the chain. The job time of the upstream job node in the job chain should be earlier than the job time of the downstream job node.

[0061] Extract the job time information and execution entity information from the above job information, and filter the historical execution statements. The historical execution statements include execution host information and execution timestamps. Match the job time information and execution entity information in the job information with the execution host information and execution timestamps in the historical execution statements to determine the historical execution statements that match the job node, and use these as the target execution statements.

[0062] Optionally, historical execution statements are filtered based on the execution host information and job time information of each job node to obtain target execution statements, including: determining historical execution statements that match the execution host information of the job node and whose execution timestamps are located between the job start time and the job end time, as target execution statements.

[0063] The execution subject information of the historical execution statement is matched with the execution host information of each job node. If a match is successful with any job node, the execution timestamp of the historical execution statement is matched with the job information of that job node to determine whether the execution timestamp of the historical execution statement is within the job start time and the job end time of that job node. If so, the historical execution statement is determined as a target execution statement in the target scenario. The target execution statements matched by multiple job nodes can form a set of target execution statements for the target scenario.

[0064] Data lineage analysis is performed on the set of target execution statements in the target scenario to obtain the data lineage analysis results. Each target execution statement includes source data and target data. Source data is the data before processing in a certain process, and target data is the data after processing by that process. For example, data A is processed to obtain data B; data A is the source data, and data B is the target data. The source data is determined to be upstream of the target data, and the target data is determined to be downstream of the source data. Each target execution statement includes a data identifier for both the source and target data. These data identifiers determine the data lineage relationship corresponding to that target execution statement.

[0065] Accordingly, constructing the data lineage analysis results of the target scenario based on the target execution statement includes: for each target execution statement, extracting the source data and target data in the target execution statement, and establishing the data lineage relationship between the source data and target data; and forming the data lineage analysis results based on the data lineage relationships corresponding to multiple target execution statements.

[0066] Based on the data identifiers of the source data and the target data, data nodes for the source data and target data are created respectively. A lineage relationship is established between these data nodes, for example, by connecting them with arrows or connecting lines to represent the data lineage relationship. The data lineage relationships corresponding to multiple target execution statements in the target scenario are then integrated to obtain the data lineage analysis results for the target scenario. For example, identical data nodes in different data lineage relationships are merged to achieve the merging of different data lineage relationships. For instance, data A has a data lineage relationship with data B, and data B has a data lineage relationship with data C. Data A, data B, and data C have sequential data lineage relationships. The data nodes of data B in the two data lineage relationships are merged to obtain the data lineage relationships of data A, data B, and data C, and so on, to obtain the data lineage analysis results.

[0067] Based on the above embodiments, the source data and target data each include one or more of the following: system data, table data, and field data. The data being processed can belong to different physical systems. A physical system can be the entity to which the data belongs; for example, a business system can be considered a physical system, or an application set can be considered a physical system. System data is used to characterize the system to which the data belongs. Any system can include multiple tables, which are in tabular form and may include field data. For example, parsing the target execution statement yields field data M from table data 1 in system 1 as the source data, and field data N from table data B in system 2 as the target data.

[0068] Accordingly, the data lineage relationships include system-level lineage relationships, table-level lineage relationships, and field-level lineage relationships. The creation process for system-level lineage relationships based on system data, table-level lineage relationships based on table data, and field-level lineage relationships based on field data is not detailed here. Consequently, the data lineage analysis results include system-level lineage relationships, table-level lineage relationships, and field-level lineage relationships.

[0069] Based on the above embodiments, this invention also provides embodiments of a data lineage analysis method in regulatory scenarios, see [link to relevant documentation]. Figure 3 , Figure 3 This is a flowchart of a data lineage analysis method provided by an embodiment of the present invention. A work chain for the regulatory scenario is created through the end-execution nodes in the regulatory scenario. The execution flow of multiple work nodes in the work chain is obtained. This execution flow includes work information of multiple work nodes in the work chain, including execution host information, work start time, and work end time. The execution host information, work start time, and work end time are matched against historical execution statements to obtain a set of target execution statements in the regulatory scenario. The data lineage analysis result for the regulatory scenario is then created using these target execution statements.

[0070] The technical solution provided by this invention determines the job chain of a target scenario, matches the target execution statement of the job chain, and eliminates historical execution statements outside the target scenario, thereby reducing interfering execution statements. By constructing the data lineage analysis result of the target scenario through the target execution statement, data lineage analysis for the target scenario is achieved, improving the accuracy of the data lineage analysis result.

[0071] Example 2

[0072] Figure 4 This is a schematic diagram of the structure of a data lineage analysis device provided in Embodiment 2 of the present invention.

[0073] like Figure 4 As shown, the device includes:

[0074] The job link determination module 210 is used to obtain the end job nodes of the target scenario and determine the job link of the target scenario based on the end job nodes;

[0075] The job information extraction module 220 is used to extract job information of multiple job nodes in the job chain, wherein the job information includes execution host information and job time information;

[0076] The statement filtering module 230 is used to filter historical execution statements based on the execution host information and job time information of each job node to obtain the target execution statement;

[0077] The data lineage construction module 240 is used to construct the data lineage analysis results of the target scenario based on the target execution statement.

[0078] Based on the above embodiments, optionally, the target scenario includes a regulatory scenario, and the end operation node of the regulatory scenario is an operation node that sends regulatory data to the regulatory equipment.

[0079] Based on the above embodiments, optionally, the job link determination module 110 is used for:

[0080] Based on the dependencies between job nodes, the multi-level upstream job nodes of the end job node are determined sequentially, and the end job node and the multi-level upstream job nodes form a job link.

[0081] Optionally, the job time information includes the job start time and job end time;

[0082] The statement filtering module 230 is used to: determine historical execution statements that match the execution host information of the job node and whose execution timestamps are located between the job start time and the job end time, and use them as target execution statements.

[0083] Based on the above embodiments, optionally, the target execution statement includes source data and target data;

[0084] The data lineage construction module 210 is used to extract source data and target data from each target execution statement, establish the data lineage relationship between the source data and target data, and form a data lineage analysis result based on the data lineage relationship corresponding to multiple target execution statements.

[0085] Based on the above embodiments, optionally, the source data and the target data each include one or more of the following: system data, table data, and field data;

[0086] Accordingly, the data lineage includes system-level lineage, table-level lineage, and field-level lineage.

[0087] The data lineage analysis device provided in this embodiment of the invention can execute the data lineage analysis method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0088] Example 3

[0089] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0090] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0091] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0092] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data lineage analysis methods.

[0093] In some embodiments, the data lineage analysis method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data lineage analysis method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data lineage analysis method by any other suitable means (e.g., by means of firmware).

[0094] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0095] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0096] Example 4

[0097] Embodiment 4 of the present invention also provides a computer-readable storage medium storing computer instructions for causing a processor to execute a data lineage analysis method, the method comprising:

[0098] Obtain the end-point job nodes of the target scenario, and determine the job chain of the target scenario based on the end-point job nodes; extract job information of multiple job nodes in the job chain, the job information including execution host information and job time information; filter historical execution statements based on the execution host information and job time information of each job node to obtain the target execution statement; construct the data lineage analysis result of the target scenario based on the target execution statement.

[0099] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0100] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0101] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0102] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0103] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0104] Example 5

[0105] Embodiment 5 of the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data lineage analysis method according to any embodiment of the present invention.

[0106] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data pedigree analysis method, characterized in that, The method comprises the following steps: obtaining an end job node of a target scene; wherein the target scene comprises a supervision scene, and the end job node of the supervision scene is a job node for sending supervision data to a supervision device; determining, based on a dependency relationship between job nodes, a plurality of upstream job nodes of the end job node in sequence; sequentially arranging the end job node and a plurality of job nodes in the plurality of upstream job nodes based on a dependency relationship or an upstream-downstream relationship to form a job link, or connecting the job nodes in series based on the dependency relationship or the upstream-downstream relationship to form the job link; verifying the job link based on a job time of each job node in the job link; wherein the job time of an upstream job node in the job link should be earlier than the job time of a downstream job node; extracting job information of a plurality of job nodes in the job link, wherein the job information comprises execution host information and job time information; filtering historical execution statements based on the execution host information and the job time information of each job node to obtain target execution statements; the target execution statements comprise source data and target data; for each target execution statement, extracting the source data and the target data in the target execution statement; the source data and the target data comprise one or more of the following: system data, table data, field data; wherein the source data and the target data belong to different physical systems; creating data nodes of the source data and the target data based on data identifiers of the source data and the target data; connecting the data nodes of the source data and the target data by means of arrows or connecting lines to establish a data lineage relationship of the source data and the target data; the data lineage relationship comprises a system-level lineage relationship, a table-level lineage relationship, and a field-level lineage relationship; integrating the data lineage relationships corresponding to a plurality of target execution statements corresponding to the target scene to obtain a data lineage analysis result of the target scene.

2. The method of claim 1, wherein, The job time information comprises a job start time and a job end time; The filtering of the historical execution statements based on the execution host information and the job time information of each job node to obtain the target execution statements comprises: determining historical execution statements that match the execution host information of the job node and have execution timestamps located between the job start time and the job end time as the target execution statements.

3. A data lineage analysis apparatus, characterized by, The method comprises the following steps: a job link determination module, configured to obtain an end job node of a target scene and determine a job link of the target scene based on the end job node; the target scene comprises a supervision scene, and the end job node of the supervision scene is a job node for sending supervision data to a supervision device; a job information extraction module, configured to extract job information of a plurality of job nodes in the job link, wherein the job information comprises execution host information and job time information; a statement filtering module, configured to filter historical execution statements based on the execution host information and the job time information of each job node to obtain target execution statements. The target execution statement includes source data and target data; The data bloodline construction module is configured to construct a data bloodline analysis result of the target scenario based on the target execution statement; The job link determination module is configured to determine, based on a dependency relationship between the job nodes, a plurality of upstream job nodes of the terminal job node in sequence; and sort the terminal job node and the plurality of job nodes in the plurality of upstream job nodes based on the dependency relationship or an upstream-downstream relationship to form a job link, or connect the job nodes in series based on the dependency relationship or the upstream-downstream relationship to form the job link. The data bloodline analysis device is further configured to verify the job link based on a job time of each job node in the job link, wherein the job time of an upstream job node in the job link should be earlier than the job time of a downstream job node. The data bloodline construction module is configured to extract, for each target execution statement, source data and target data in the target execution statement, wherein the source data and the target data include one or more of system data, table data, and field data, and the source data and the target data belong to different physical systems; create data nodes of the source data and the target data based on data identifiers of the source data and the target data; and connect the data nodes of the source data and the target data by arrows or connecting lines to establish a data bloodline relationship between the source data and the target data, wherein the data bloodline relationship includes a system-level bloodline relationship, a table-level bloodline relationship, and a field-level bloodline relationship; and integrate data bloodline relationships corresponding to a plurality of target execution statements corresponding to the target scenario to obtain a data bloodline analysis result of the target scenario.

4. The apparatus of claim 3, wherein, The job time information includes a job start time and a job end time. The statement screening module is configured to determine a historical execution statement that matches execution host information of the job node and has an execution timestamp located between the job start time and the job end time as a target execution statement.

5. An electronic device, comprising: The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the data bloodline analysis method of any one of claims 1-2.

6. A computer readable storage medium characterized by The computer readable storage medium stores computer instructions for causing the processor to execute the data bloodline analysis method of any one of claims 1-2 when executed.

7. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by a processor, implements the data bloodline analysis method according to any one of claims 1-2.

Images