Data blood relationship determination method and device, computer equipment and storage medium

Abstract

The invention provides a data consanguinity determination method and device, computer equipment and a storage medium. The method comprises the steps of obtaining a target SQL statement corresponding to a target batch processing task; performing data link analysis on the target SQL statement to generate node information and node association information; and generating a data link map based on the node information and the node association information. By implementing the method disclosed by the invention, the relatively comprehensive node information and node association information can be quickly and accurately obtained by performing data link analysis on the standard SQL statement, so that the indication accuracy of the obtained data link map on the data blood relationship is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of data governance technology, and in particular to a method, apparatus, computer equipment and storage medium for determining data lineage. Background Art

[0002] As the functions of a company's business and financial systems continue to grow and expand, the interconnections between system data have become increasingly complex. For such systems that carry massive amounts of data and have complex flows, when assessing the impact of changes in requirements or functional modifications, if sufficient data link analysis is not performed, the scope of impact may be insufficiently assessed. In terms of data governance and operation and maintenance, whether it is monitoring and analyzing data quality or formulating data processing plans for operation and maintenance issues, an in-depth and detailed analysis of the system's data flow is required to ensure the smooth progress of related work. Data flow analysis is also called data lineage analysis.

[0003] The related technologies are incomplete and inaccurate when performing data lineage analysis. Summary of the invention

[0004] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the purpose of the present invention is to propose a data lineage determination method, device, computer equipment and storage medium, which can perform data link analysis by benchmarking SQL statements, quickly and accurately obtain more comprehensive node information and node association information, thereby ensuring the accuracy of the obtained data link map in indicating data lineage relationships.

[0006] To achieve the above-mentioned purpose, the data lineage determination method proposed in the first aspect of the present disclosure includes:

[0007] Get the target SQL statement corresponding to the target batch task;

[0008] Performing data link analysis on the target SQL statement to generate node information and node association information;

[0009] A data link map is generated based on the node information and the node association information.

[0010] To achieve the above-mentioned purpose, the data lineage determination device proposed in the second aspect of the present disclosure includes:

[0011] The acquisition module is used to obtain the target SQL statement corresponding to the target batch processing task;

[0012] A first generating module, used for performing data link analysis on the target SQL statement to generate node information and node association information;

[0013] The second generating module is used to generate a data link map based on the node information and the node association information.

[0014] The computer device proposed in the third aspect embodiment of the present disclosure includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the data lineage determination method proposed in the first aspect embodiment of the present disclosure is implemented.

[0015] The fourth aspect embodiment of the present disclosure proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the data lineage determination method proposed in the first aspect embodiment of the present disclosure is implemented.

[0016] The fifth aspect embodiment of the present disclosure proposes a computer program product. When the instructions in the computer program product are executed by a processor, the data lineage determination method proposed in the first aspect embodiment of the present disclosure is executed.

[0017] The data lineage determination method, device, computer equipment and storage medium provided by the present disclosure obtain the target SQL statement corresponding to the target batch processing task; perform data link analysis on the target SQL statement to generate node information and node association information; and generate a data link map based on the node information and node association information. Thus, it is possible to quickly and accurately obtain relatively comprehensive node information and node association information by performing data link analysis on the benchmark SQL statement, thereby ensuring the accuracy of the data link map in indicating the data lineage relationship.

[0018] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description or learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 It is a flowchart of a method for determining data lineage proposed in an embodiment of the present disclosure;

[0021] Figure 2 is a flowchart of a method for determining data lineage according to another embodiment of the present disclosure;

[0022] Figure 3 is a flow chart of data link tracking and detection proposed in the present disclosure;

[0023] Figure 4 is a schematic diagram of the structure of a data lineage determination device proposed in an embodiment of the present disclosure;

[0024] Figure 5 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present disclosure, and are not to be construed as limitations of the present disclosure. On the contrary, the embodiments of the present disclosure include all changes, modifications, and equivalents that fall within the spirit and connotation of the appended claims.

[0026] Figure 1 It is a flowchart of a method for determining data lineage proposed in an embodiment of the present disclosure.

[0027] It should be noted that the executor of the data lineage determination method of this embodiment is a data lineage determination device, which can be implemented by software and / or hardware. The device can be configured in a computer device, and the computer device can include but is not limited to a terminal, a server, etc. For example, the terminal can be a mobile phone, a handheld computer, etc.

[0028] like Figure 1 As shown, the data lineage determination method includes:

[0029] S101: Obtain a target SQL statement corresponding to a target batch processing task.

[0030] The target batch processing task refers to the data batch processing task corresponding to any business system function.

[0031] The target SQL statement refers to the SQL statement involved in the target batch processing task.

[0032] In the disclosed embodiment, when a target SQL statement corresponding to a target batch processing task is obtained, a reliable analysis object can be provided for determining data lineage relationships.

[0033] S102: Perform data link analysis on the target SQL statement to generate node information and node association information.

[0034] The node information may refer to the nodes in the data link determined by the target SQL statement through data link analysis.

[0035] The node association information may be used to indicate the association relationship between nodes.

[0036] Optionally, in some embodiments, the node information includes: current node ID, data table identifier, field identifier, parent node ID, and the level of the current node; the node association information includes: start node ID, end node ID, task identifier, library table association condition, association type, and association status. In this way, the indication effect of the obtained node information and node association information can be effectively improved, and the practicality of the obtained node information and node association information in the process of determining data lineage can be ensured.

[0037] S103: Generate a data link graph based on the node information and node association information.

[0038] Among them, the data link graph refers to a graph generated based on node information and node association information to indicate data lineage relationships, and the graph may include association relationships between different data.

[0039] In this embodiment, the target SQL statement corresponding to the target batch task is obtained; data link analysis is performed on the target SQL statement to generate node information and node association information; and a data link map is generated based on the node information and node association information. Thus, data link analysis can be performed on the benchmark SQL statement to quickly and accurately obtain relatively comprehensive node information and node association information, thereby ensuring the accuracy of the data link map in indicating the data lineage relationship.

[0040] Optionally, in some embodiments, a task-related table can be determined based on node information; randomly sample task-related table production data; and determine and correct the association type based on the production data. Thus, after generating the data link map, the node association type can be checked and corrected to effectively improve the robustness of the data lineage determination process.

[0041] The task-related table refers to a table related to the target batch processing task.

[0042] Optionally, in some embodiments, when determining and correcting the association type based on production data, it can be: determining whether the dependent fields of the node association information are consistent based on the production data to determine the target association type; and correcting the association type based on the target association type. Thus, the target association type can be determined based on whether the dependent fields are consistent, and the correction processing of the association type can be implemented based on the target association type to ensure the reliability of the association type.

[0043] The target association type refers to an association type determined based on whether dependent fields are consistent, for example, it can be a direct reference or a derivation.

[0044] Optionally, in some embodiments, a third matching result between the target association type and the association type may be determined, and the association state may be updated based on the third matching result. Thus, the association state may be updated based on the third matching result between the target association type and the association type to ensure the reliability of the finally obtained association state.

[0045] The third matching result may be used to indicate whether the target association type is consistent with the content indicated by the association type.

[0046] For example, in the embodiments of the present disclosure, when the association status is updated based on the matching result, when the association type is a null value, or when the target association type is inconsistent with the association type, the target association type is used as the new association type.

[0047] Figure 2 It is a flowchart of a method for determining data lineage proposed in another embodiment of the present disclosure.

[0048] like Figure 2 As shown, the data lineage determination method includes:

[0049] S201: Obtain a target SQL statement corresponding to a target batch processing task.

[0050] S202: Perform data link analysis on the target SQL statement to generate node information and node association information.

[0051] The description of S201 and S202 can be specifically referred to the above embodiment, which will not be repeated here.

[0052] S203: Determine a first matching result between the target SQL statement and the node information, and a second matching result between the target SQL statement and the node association information.

[0053] The first matching result can be used to indicate whether the content indicated by the node information matches the relevant content in the target SQL statement, and the second matching result can be used to indicate whether the node association information matches the relevant content in the target SQL statement.

[0054] It is understandable that when performing data link analysis on the target SQL statement, some node information and node association information may be omitted. Therefore, in the embodiment of the present disclosure, a first matching result between the target SQL statement and the node information, as well as a second matching result between the target SQL statement and the node association information can be determined, thereby providing a reliable execution basis for the subsequent adjustment process.

[0055] S204: Adjust the node information according to the first matching result.

[0056] For example, in the embodiment of the present disclosure, when adjusting the node information according to the first matching result, the user may manually supplement the missing node information.

[0057] S205: Adjust the node association information according to the second matching result.

[0058] For example, in the embodiment of the present disclosure, when the node association information is adjusted according to the second matching result, the user may manually supplement the missing node association information.

[0059] S206: Generate a data link map based on the adjusted node information and the adjusted node association information.

[0060] That is, in the embodiment of the present disclosure, after generating the node information and the node association information, the first matching result between the target SQL statement and the node information and the second matching result between the target SQL statement and the node association information can be determined; the node information is adjusted according to the first matching result; the node association information is adjusted according to the second matching result; and a data link map is generated based on the adjusted node information and the adjusted node association information. Thus, the node information and the node association information can be adjusted in time before the data link map is generated to ensure the reliability and accuracy of the obtained data link map.

[0061] In this embodiment, by determining a first matching result between a target SQL statement and node information, and a second matching result between a target SQL statement and node association information; adjusting the node information according to the first matching result; adjusting the node association information according to the second matching result; and generating a data link map based on the adjusted node information and the adjusted node association information. Thus, the node information and the node association information can be adjusted in time before the data link map is generated to ensure the reliability and accuracy of the obtained data link map.

[0062] Based on the above embodiments, the present disclosure analyzes the table-level and field-level association relationships of system functional data, distinguishes whether the field association relationships are direct or derived, and uses a link map to display the flow of various types of data in the system. At the same time, it supports the use of production data to detect the link map and verify the direct field and derived relationships, providing a reference for data governance, operation and maintenance plan formulation, R&D impact range assessment and other work.

[0063] For example, Figure 3 As shown, Figure 3 The following is a flow chart of data link tracking and detection according to the present disclosure, which includes the following steps:

[0064] 1. Divide SQL code by task:

[0065] The functions of data processing systems are mostly reflected in background tasks, and SQL is divided by task to provide to data analysis tools;

[0066] 2. Automatically analyze data links and generate data link nodes and connection information:

[0067] Transform SQLLineage to analyze the SQL under the task (support single SQL, multiple SQL, and multi-level SQL chain analysis of stored procedures under a single task. By recording the changes in field names under multiple SQLs, the direct reference or derivation relationship between fields with different names can be recorded). Obtain the task-data table and the blood relationship between fields in different data tables:

[0068] (1) Generate node information:

[0069] Generate node numbers for fields according to field lineage relationships and determine parent nodes. The hierarchical information is shown in Table 1. 005 is the first-level node id, 002 is the second-level node id, name reflects the library table and field information, and level is the level.

[0070] Table 1

[0071] id name parent_id level 005-002 tablename.fieldname 0005 2

[0072] (2) Generate node association information, as shown in Table 2:

[0073] Record the node order for drawing connection lines. Also record the task code to clarify the relationship between nodes and tasks.

[0074] conditions are the database-table association conditions, used for display.

[0075] type is the association type: 01-direct reference; 02-derived.

[0076] status is the status: 0-initial; 1-correct; 2-type inconsistent: the new type is derived; 3-type inconsistent: the new type is a direct reference.

[0077] Table 2

[0078] start end taskcode conditions type status 003 001 zrjPniGenerHandler For example, tablename.fieldname = '05' 01 1

[0079] 3. Manual import supplement:

[0080] In addition to automatic generation, manual link information supplement can be performed for special SQL writing methods.

[0081] 4. Generate data link map:

[0082] After selecting a task, the relationship between databases is displayed. After selecting a target table field, the field association relationship is displayed below. In the field link diagram, the type of association relationship of each field is displayed by color (different colors are used to mark direct references and derivatives).

[0083] 5. Confirm the task related table:

[0084] According to the above configuration table information, obtain the task-related data table and the data random sampling condition SQL.

[0085] 6. Random sampling task related table production data:

[0086] Sample SQL statements based on data from all tasks and sample production data from related tables.

[0087] 7. Determine and correct the association type based on production data:

[0088] (1) Determine whether dependent fields are consistent field by field, and support manual adjustment of association types:

[0089] ① If they are consistent, it is determined to be a direct quotation;

[0090] ②If inconsistent, it is judged as derived data.

[0091] (2) Determine whether the current dependency has been classified:

[0092] ① Existing categories, determine whether it is the same as the existing categories

[0093] 1) Yes, the determination and correction are completed;

[0094] 2) No, record the classification difference data link connection information for identification in the field link map of the function interface, and use an icon to identify the connection line. An example is [Please confirm: Type is inconsistent: the new type is derived].

[0095] ②No, this is the first classification and this judgment and correction is completed.

[0096] Based on the above embodiments, the present disclosure uses system functions as units to establish data table-level and field-level link relationships and intuitively display them, which can distinguish between direct references and derived types. Through the reference and derived relationships of fields, it will provide a direct reference basis for system production data quality monitoring (reference relationship fields are strictly equal, and derived relationship fields have size associations) and automated testing work (providing integration-level and system-level verification points, and automated test assertions are more accurate and efficient). 2. Support the use of production data to detect data link accuracy and ensure the continued effectiveness of data links.

[0097] Figure 4 It is a structural diagram of a data lineage determination device proposed in an embodiment of the present disclosure.

[0098] like Figure 4 As shown, the data lineage determination device 40 comprises:

[0099] The acquisition module 401 is used to acquire the target SQL statement corresponding to the target batch processing task;

[0100] The first generating module 402 is used to perform data link analysis on the target SQL statement and generate node information and node association information;

[0101] The second generating module 403 is used to generate a data link graph based on the node information and the node association information.

[0102] It should be noted that the aforementioned explanation of the data lineage determination method is also applicable to the data lineage determination device of this embodiment and will not be repeated here.

[0103] In this embodiment, the target SQL statement corresponding to the target batch task is obtained; data link analysis is performed on the target SQL statement to generate node information and node association information; and a data link map is generated based on the node information and node association information. Thus, data link analysis can be performed on the benchmark SQL statement to quickly and accurately obtain relatively comprehensive node information and node association information, thereby ensuring the accuracy of the data link map in indicating the data lineage relationship.

[0104] Figure 5 A block diagram of an exemplary computer device suitable for implementing embodiments of the present disclosure is shown. Figure 5 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0105] like Figure 5 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0106] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0107] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0108] The memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 Not shown, often called a "hard drive").

[0109] although Figure 5 Not shown, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (Compact Disc Read Only Memory; hereinafter referred to as: CD-ROM), a digital versatile disc read only memory (Digital Video Disc Read Only Memory; hereinafter referred to as: DVD-ROM) or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0110] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described in the present disclosure.

[0111] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), one or more devices that enable a human body to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the data lineage determination method mentioned in the above embodiment.

[0113] In order to implement the above embodiments, the present disclosure also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data lineage determination method proposed in the above embodiments of the present disclosure is implemented.

[0114] In order to implement the above embodiments, the present disclosure also proposes a computer program product. When the instruction processor in the computer program product is executed, the data lineage determination method proposed in the above embodiments of the present disclosure is executed.

[0115] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0116] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0117] It should be noted that, in the description of the present disclosure, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0118] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present disclosure belong.

[0119] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0120] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0121] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0122] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0123] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0124] Although the embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A method for determining data lineage, characterized in that: include: Get the target SQL statement corresponding to the target batch task; Performing data link analysis on the target SQL statement to generate node information and node association information; A data link map is generated based on the node information and the node association information.

2. The method according to claim 1, characterized in that The generating of a data link graph based on the node information and the node association information includes: Determine a first matching result between the target SQL statement and the node information, and a second matching result between the target SQL statement and the node association information; Adjusting the node information according to the first matching result; adjusting the node association information according to the second matching result; The data link map is generated based on the adjusted node information and the adjusted node association information.

3. The method according to claim 1, characterized in that in, The node information includes: current node ID, data table ID, field ID, parent node ID and the level of the current node; The node association information includes: a start node ID, an end node ID, a task identifier, a library table association condition, an association type, and an association status.

4. The method according to claim 3, characterized in that The method further comprises: Based on the node information, determine a task-related table; Randomly sampling the task-related table production data; The association type is determined and corrected based on the production data.

5. The method according to claim 4, characterized in that The determining and correcting the association type according to the production data includes: Determining whether the dependent fields of the node association information are consistent according to the production data to determine the target association type; The association type is corrected based on the target association type.

6. The method according to claim 5, characterized in that The method further comprises: Determine a third matching result between the target association type and the association type; Based on the third matching result, the association status is updated.

7. A data lineage determination device, characterized in that: include: The acquisition module is used to obtain the target SQL statement corresponding to the target batch processing task; A first generating module, used for performing data link analysis on the target SQL statement to generate node information and node association information; The second generating module is used to generate a data link map based on the node information and the node association information.

8. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6.

Images