Data lineage acquisition method, device, equipment, and computer program product

Abstract

An embodiment of the present invention discloses a method, device, equipment and computer program product for obtaining and identifying data lineage, which relates to the field of big data technology, and includes: obtaining a data lineage query instruction input by a user, where the data lineage query instruction contains the data to be queried; querying the data to be queried through a recursive algorithm to obtain associated data that has a dependency relationship with the data to be queried, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship; determining the data lineage of the data to be queried according to the associated data and the dependency relationship, and displaying the data lineage in a specified format. By using a recursive algorithm to obtain associated data that has a direct dependency relationship or an indirect dependency relationship with the data to be queried, it will not cause the lack of intermediate links when determining the data association relationship, so as to display a comprehensive data lineage according to the user's data query requirements.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data technology, and in particular, to a method, device, equipment, and computer program product for obtaining data lineage relationships. Background Art

[0002] With the development and application of big data technology, data is increasingly regarded as an asset by various industries. One of the most important links in the collation of this intangible asset is to clarify the lineage dependencies of each table and field in the ETL, that is, the data warehouse technology process.

[0003] However, most current lineage analyses can only go from the target table to the final dependent tables and fields, and will ignore intermediate temporary tables and temporary fields, resulting in the absence of intermediate links when determining data association relationships, and thus it is impossible to display comprehensive data lineage relationships according to the user's data query requirements. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device, equipment, and computer program product for obtaining data lineage relationships to achieve obtaining comprehensive data lineage relationships.

[0005] In a first aspect, an embodiment of the present invention provides a method for obtaining data lineage relationships, including: obtaining a data lineage query instruction input by a user, where the data lineage query instruction includes data to be queried;

[0006] Querying the data to be queried through a recursive algorithm to obtain associated data having a dependency relationship with the data to be queried, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship;

[0007] Determining the data lineage relationship of the data to be queried according to the associated data and the dependency relationship, and displaying the data lineage relationship in a specified format, where the data lineage relationship is used to identify the associated data and corresponding relationships involved from the generation to the invalidation of the data to be queried.

[0008] In a second aspect, an embodiment of the present invention provides a device for obtaining data lineage relationships, including:

[0009] A data query instruction obtaining module, configured to obtain a data lineage query instruction input by a user, where the data lineage query instruction includes data to be queried;

[0010] An associated data obtaining module, configured to query the data to be queried through a recursive algorithm to obtain associated data having a dependency relationship with the data to be queried, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship;

[0011] A blood relationship display module, configured to determine the data blood relationship of the data to be queried according to the associated data and the dependency relationship, and display the data blood relationship in a specified format, where the data blood relationship is used to identify the associated data and the corresponding relationship involved from the generation to the invalidation of the data to be queried.

[0012] In a third aspect, an embodiment of the present invention provides an electronic device, which includes:

[0013] One or more processors;

[0014] A storage device for storing one or more programs,

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0017] The technical solution of the embodiment of the present invention obtains the associated data that has a direct or indirect dependency relationship with the data to be queried through a recursive algorithm, without causing the lack of intermediate links when determining the data association relationship, so as to display a comprehensive data blood relationship according to the user's query data requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 is a flowchart of the data blood relationship acquisition method provided by Embodiment 1 of the present invention;

[0020] Figure 2 is a schematic diagram of the blood relationship display in a graphical format provided by Embodiment 1 of the present invention;

[0021] Figure 3 is a flowchart of the data blood relationship acquisition method provided by Embodiment 2 of the present invention;

[0022] Figure 4 is a schematic structural diagram of the data blood relationship acquisition device provided by Embodiment 3 of the present invention;

[0023] Figure 5 is a schematic structural diagram of an electronic device provided by Embodiment 4 of the present invention. Detailed implementation manners

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention.

[0025] In addition, it should be noted that, for the sake of convenience of description, only the parts related to the present invention rather than all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the process can be terminated, but there may also be additional steps not included in the drawings. The process can correspond to a method, a software implementation, a hardware implementation, etc. The acquisition, storage, use, processing, etc. of data in the technical solution of the present application all comply with the relevant regulations of national laws and regulations.

[0026] Embodiment 1

[0027] Figure 1 is a flowchart of the method for obtaining data lineage provided by the embodiment of the present invention. This embodiment is applicable to the situation of obtaining comprehensive data lineage. This method can be executed by the data lineage acquisition device in the embodiment of the present invention, and this device can be implemented in a software and / or hardware manner. As Figure 1 shown, the method specifically includes the following operations:

[0028] Step S101, obtain a data lineage query instruction input by a user, where the data lineage query instruction includes data to be queried.

[0029] Optionally, before obtaining the data lineage query instruction input by the user, it further includes: obtaining original data; storing the original data in a specified database according to a preset path, where the preset path includes a path composed of a system name, a table name, and a field name.

[0030] Specifically, before obtaining the data lineage query instruction input by the user in this embodiment, it is necessary to first clarify the object of data query, that is, where to query. Therefore, before formally preparing for the query, the original data input by the user or developer in advance can be obtained. The original data can specifically be engineering data or development data. In this embodiment, the specific type of the original data is not limited. As long as it is related to the query scenario and is convenient for the user to query later, it is within the protection scope of this application, and this embodiment will not elaborate on it. After obtaining the original data, it will be stored in the specified database according to the preset path. For example, the original data is stored in Table Data1 under System X, and specifically in field ab of Data1. The preset path can also only include the system name and table name. In this case, the specific stored field is not limited, or the preset path can also only include the system name. In this case, the specific table and field are not limited. Of course, this is only an example in this embodiment, and the specific preset path is not limited.

[0031] Optionally, after obtaining the data lineage query instruction input by the user, it further includes: detecting the data to be queried; determining that the data to be queried is located in the specified database according to the inspection result.

[0032] Among them, in this embodiment, after obtaining the data lineage query instruction, an existence verification detection will also be performed on the data to be queried, that is, to determine whether the data to be queried is within the range that the user can query. Since data is stored in the specified database in advance, and the field names, table names, or system names of the stored data are displayed through the human-computer interaction page, the user can select the data to be queried by clicking, or directly input the data to be queried from the query entry. However, since the data in the database changes in real time, after obtaining the data lineage query instruction input by the user by clicking or writing, it will be further determined whether the data to be queried is located in the specified database. Only when it is determined to exist will the subsequent data lineage relationship query process be carried out.

[0033] Step S102, query the data to be queried through a recursive algorithm to obtain associated data that has a dependency relationship with the data to be queried.

[0034] Optionally, the data to be queried includes a target system name, a combination of the target system name and the target table name, or a combination of the target system name, the target table name, and the target field name.

[0035] Specifically, the data to be queried in this embodiment may only include the target system name, that is, to query the lineage of data in the specified system; or may include both the system name and the target table name, that is, to query the lineage of data in the specified target table in the specified system; or may include both the system name, the target table name, and the target field name, that is, specifically to query the lineage of data in the specified target field under the specified target table in the specified system. Therefore, it can be seen from the above content that when the content in the data to be queried is more detailed, the data range to be queried is smaller. Of course, in this embodiment, only examples are given, and the specific content of the data to be queried is not limited. As long as the associated data that has a direct or indirect dependency relationship with the data to be queried can be obtained, it is within the protection scope of this application, and it will not be elaborated in this embodiment.

[0036] Optionally, a recursive algorithm is used to query the data to be queried to obtain the associated data that has a dependency relationship with the data to be queried, including: recursively querying level by level from the specified database according to the data to be queried to obtain the first associated data that has a direct dependency relationship with the data to be queried; and obtaining the second associated data that has an indirect dependency relationship with the data to be queried based on the first associated data.

[0037] In a specific implementation, in this embodiment, a recursive algorithm is specifically used to query the data to be queried in the specified database. Querying the data to be queried through a recursive algorithm is to continuously recurse until the final dependent field is found. For example, when the combination of the target system name, the target table name, and the target field name of the data to be queried is determined, the target field is queried from the target table name under the target system name to obtain the first associated data that has a direct dependency relationship with the target field, for example, the first-level source table and the first-level source table field; then, based on the first associated data, the second associated data that has an indirect dependency relationship with the target field is obtained, for example, the second-level source table and the second-level source table field, etc. Among them, when querying, a cross-system method can be specifically used for querying, so this embodiment can be applied to the scenario of a distributed system. And because it is searched layer by layer through a recursive algorithm, the lack of intermediate links will not be caused, ensuring the comprehensiveness of data search.

[0038] Step S103, determine the data lineage of the data to be queried according to the associated data and the dependency relationship, and display the data lineage in a specified format.

[0039] Optionally, determine the data lineage based on the associated data and the dependencies, and display the data lineage in a specified format, including: determining a mapping rule according to the first associated data and the direct dependency, and determining a filtering condition according to the second associated data and the indirect dependency; determining the data lineage according to the first associated data, the second associated data, the mapping rule, and the filtering condition; and displaying the data lineage in a graphical format or a tabular format.

[0040] Specifically, in this embodiment, a mapping rule can be determined according to the first associated data and the direct dependency, a filtering condition can be determined according to the second associated data and the indirect dependency, and the data lineage of the data to be queried can be obtained specifically according to the above information, and can be displayed in multiple ways, such as Figure 2 shown, which is a display method in graphical format, or displayed in a tabular format as shown in Table 1 below:

[0041] Table 1

[0042]

[0043] Among them, for the data lineage shown in Table 1, the data to be queried included in the data lineage query instruction is specifically: the target system is X, the target table is Data1, and the target field is ab. Then, recursive hierarchical queries are performed from the specified database according to the data to be queried to obtain the first associated data that has a direct dependency with the data to be queried: the first-level source table Data1, the first-level source table field ac, and the mapping rule (first-level) corresponding to the direct dependency is specifically: Rule 1. Based on the first associated data, recursive hierarchical queries are performed from the specified database to obtain the second associated data that has an indirect dependency with the first associated data: the second-level source table Data2, the second-level source table field ad, and the filtering condition corresponding to the indirect dependency is specifically: Condition 1. Due to space limitations, only an example is given in Table 1, and the specific content included in the data lineage is not limited.

[0044] The technical solution of the embodiment of the present invention obtains the associated data that has a direct dependency or an indirect dependency with the data to be queried through a recursive algorithm, without causing the lack of intermediate links when determining the data association relationship, so as to display the comprehensive data lineage according to the user's data query requirements.

[0045] Embodiment 2

[0046] Figure 3It is a flowchart of the method for obtaining data lineage provided by an embodiment of the present invention. This embodiment is based on the above embodiment. After presenting the data lineage in a specified format, this embodiment further includes: verifying the data lineage, and when it is determined that the data lineage is in an abnormal state according to the verification result, an alarm prompt is given. The method steps specifically include the following operations:

[0047] Step S201: Obtain a data lineage query instruction input by the user, where the data lineage query instruction contains the data to be queried.

[0048] Optionally, before obtaining the data lineage query instruction input by the user, it further includes: obtaining the original data; storing the original data in a specified database according to a preset path, where the preset path includes a path composed of a system name, a table name, and a field name.

[0049] Optionally, after obtaining the data lineage query instruction input by the user, it further includes: detecting the data to be queried; determining that the data to be queried is located in the specified database according to the inspection result.

[0050] Step S202: Query the data to be queried through a recursive algorithm to obtain associated data that has a dependency relationship with the data to be queried.

[0051] Optionally, the data to be queried includes a target system name, a combination of a target system name and a target table name, or a combination of a target system name, a target table name, and a target field name.

[0052] Optionally, querying the data to be queried through a recursive algorithm to obtain associated data that has a dependency relationship with the data to be queried includes: recursively querying step by step from the specified database according to the data to be queried to obtain first associated data that has a direct dependency relationship with the data to be queried; obtaining second associated data that has an indirect dependency relationship with the data to be queried based on the first associated data.

[0053] Step S203: Determine the data lineage of the data to be queried according to the associated data and the dependency relationship, and present the data lineage in a specified format.

[0054] Optionally, determining the data lineage according to the associated data and the dependency relationship, and presenting the data lineage in a specified format includes: determining a mapping rule according to the first associated data and the direct dependency relationship, and determining a filtering condition according to the second associated data and the indirect dependency relationship; determining the data lineage according to the first associated data, the second associated data, the mapping rule, and the filtering condition; presenting the data lineage in a graphical format or a table format.

[0055] Step S204: Check the data lineage. When it is determined that the data lineage is in an abnormal state according to the check result, an alarm prompt is given.

[0056] Specifically, in this embodiment, after the data lineage is determined and the data lineage is displayed in a specified format, the data lineage is also checked to check whether the displayed data lineage is in an abnormal state. Specifically, it is checked whether there are obvious errors or inconsistencies with the actual situation in the data lineage. For example, if there are blank data or garbled characters in the displayed data lineage, it is determined that the displayed data lineage has obvious errors; when it is determined during the previous detection that the data to be queried is located in the specified database, but the data lineage of the queried data is blank data, then it is determined that it does not match the actual situation. In the above cases, it is determined that the check fails. Of course, only examples are given in this embodiment, and the specific form of the abnormal state is not limited.

[0057] It should be noted that when it is determined that the displayed data lineage is abnormal, an alarm prompt will be sent in time. Specifically, the alarm prompt can be in the form of pictures or voices, so that maintenance personnel can timely repair the equipment or software configuration according to the alarm prompt, thereby improving the accuracy of obtaining the data lineage. Of course, only pictures or voices are taken as examples to illustrate the alarm prompt in this embodiment. As long as it can play a prompting role for users, it is within the protection scope of this application, and it will not be elaborated in this embodiment.

[0058] The technical solution of the embodiment of the present invention obtains associated data that has a direct or indirect dependency relationship with the data to be queried through a recursive algorithm, and will not cause the lack of intermediate links when determining the data association relationship, so as to display a comprehensive data lineage according to the user's data query requirements. And by checking the obtained data lineage, an alarm prompt is given when it is determined that the data lineage is abnormal according to the check result, so as to improve the efficiency and accuracy of obtaining the data lineage.

[0059] Embodiment Three

[0060] Figure 4 FIG. 3 is a structural schematic diagram of a data lineage acquisition device provided by an embodiment of the present invention. The device includes: a data query instruction acquisition module 310, an associated data acquisition module 320, and a lineage display module 330.

[0061] The data query instruction acquisition module 310 is used to acquire a data lineage query instruction input by a user, and the data lineage query instruction includes the data to be queried;

[0062] The associated data acquisition module 320 is used to query the to-be-query data through a recursive algorithm to obtain associated data that has a dependency relationship with the to-be-query data, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship;

[0063] The blood relationship display module 330 is used to determine the data blood relationship of the to-be-query data according to the associated data and the dependency relationship, and display the data blood relationship in a specified format, where the data blood relationship is used to identify the associated data and the corresponding relationship involved from the generation to the invalidation of the to-be-query data.

[0064] Optionally, the device further includes: a specified database determination module, which is used to obtain the original data; store the original data in a specified database according to a preset path, where the preset path includes a path composed of a system name, a table name, and a field name.

[0065] Optionally, the device further includes a detection module, which is used to detect the to-be-query data;

[0066] Determine that the to-be-query data is located in the specified database according to the inspection result.

[0067] Optionally, the to-be-query data includes a target system name, a combination of a target system name and a target table name, or a combination of a target system name, a target table name, and a target field name.

[0068] Optionally, the associated data acquisition module is used for:

[0069] Perform recursive hierarchical queries from the specified database according to the to-be-query data to obtain first associated data that has a direct dependency relationship with the to-be-query data;

[0070] Based on the first associated data, obtain second associated data that has an indirect dependency relationship with the to-be-query data.

[0071] Optionally, the blood relationship display module is used for:

[0072] Determine a mapping rule according to the first associated data and the direct dependency relationship, and determine a filtering condition according to the second associated data and the indirect dependency relationship;

[0073] Determine the data blood relationship according to the first associated data, the second associated data, the mapping rule, and the filtering condition;

[0074] Display the data blood relationship in a graphical format or a table format.

[0075] Optionally, the device further includes a verification module, which is used to verify the data blood relationship;

[0076] When it is determined according to the verification result that the data lineage relationship is in an abnormal state, an alarm prompt is given.

[0077] The above device can execute the data lineage relationship acquisition method provided by any embodiment of the present invention, and has corresponding function modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference may be made to the method provided by any embodiment of the present invention.

[0078] Embodiment 4

[0079] Figure 5 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. Figure 5 It shows a block diagram of an exemplary electronic device 412 suitable for implementing the embodiments of the present invention. Figure 5 The shown electronic device 412 is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0080] As Figure 5 shown, the electronic device 412 appears in the form of a general-purpose computing device. The components of the electronic device 412 may include, but are not limited to: one or more processors 416, a memory 428, and a bus 418 connecting different system components (including the memory 428 and the processor 416).

[0081] The bus 418 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the multiple bus structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0082] The electronic device 412 typically includes a variety of computer system-readable media. These media can be any available media accessible by the electronic device 412, including volatile and non-volatile media, removable and non-removable media.

[0083] The memory 428 is used to store instructions. The memory 428 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 430 and / or cache memory 432. The electronic device 412 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 434 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, commonly referred to as a "hard disk drive"). Although Figure 5Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) can be provided. In these cases, each drive can be connected to the bus 418 through one or more data medium interfaces. The memory 428 can include at least one program product having a set (such as at least one) of program modules configured to perform the functions of the embodiments of the present invention.

[0084] A program / utility 440 having a set (at least one) of program modules 442 can be stored, for example, in the memory 428. Such program modules 442 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. The implementation of a network environment may be included in each or some combination of these examples. The program modules 442 generally perform the functions and / or methods in the embodiments described in the present invention.

[0085] The electronic device 412 can also communicate with one or more external devices 414 (such as a keyboard, a pointing device, a display 424, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 412, and / or communicate with any device that enables the electronic device 412 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 422. Also, the electronic device 412 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 420. As shown in the figure, the network adapter 420 communicates with other modules of the electronic device 412 through the bus 418. It should be understood that although Figure 5 not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0086] The processor 416 executes various functional applications and data processing by running the instructions stored in the memory 428, for example, implementing the data lineage acquisition method provided by the embodiments of the present invention: obtaining a data lineage query instruction input by a user, where the data lineage query instruction includes data to be queried; querying the data to be queried through a recursive algorithm to obtain associated data having a dependency relationship with the data to be queried, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship; determining the data lineage of the data to be queried according to the associated data and the dependency relationship, and displaying the data lineage in a specified format, where the data lineage is used to identify the associated data and corresponding relationships involved from the generation to the invalidation of the data to be queried.

[0087] Embodiment 5

[0088] The embodiments of the present invention provide a computer program product, including a computer program, which when executed by a processor, implements the data lineage acquisition method provided by all the inventive embodiments of the present application: obtaining a data lineage query instruction input by a user, where the data lineage query instruction includes data to be queried; querying the data to be queried through a recursive algorithm to obtain associated data having a dependency relationship with the data to be queried, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship; determining the data lineage of the data to be queried according to the associated data and the dependency relationship, and displaying the data lineage in a specified format, where the data lineage is used to identify the associated data and corresponding relationships involved from the generation to the invalidation of the data to be queried.

[0089] Any combination of one or more computer-readable media may be employed. The computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, the computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0090] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0091] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0092] The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0093] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments may be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for obtaining data lineage, characterized in that, Including: Obtain a data lineage query instruction input by the user, where the data lineage query instruction contains the data to be queried; Query the data to be queried through a recursive algorithm to obtain associated data that has a dependency relationship with the data to be queried, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship; Determine the data lineage relationship of the data to be queried based on the associated data and the dependency relationship, and display the data lineage relationship in a specified format, where the data lineage relationship is used to identify the associated data and the corresponding relationship involved from the generation to the invalidation of the data to be queried; Before obtaining the data lineage query instruction input by the user, it further includes: obtaining original data; storing the original data in a specified database according to a preset path, where the preset path includes a path composed of a system name, a table name, and a field name; The querying the data to be queried through a recursive algorithm to obtain associated data that has a dependency relationship with the data to be queried includes: recursively querying the data to be queried from the specified database step by step to obtain first associated data that has a direct dependency relationship with the data to be queried; obtaining second associated data that has an indirect dependency relationship with the data to be queried based on the first associated data; The determining the data lineage relationship of the data to be queried based on the associated data and the dependency relationship, and displaying the data lineage relationship in a specified format includes: determining a mapping rule according to the first associated data and the direct dependency relationship, and determining a filtering condition according to the second associated data and the indirect dependency relationship; determining the data lineage relationship of the data to be queried according to the first associated data, the second associated data, the mapping rule, and the filtering condition; displaying the data lineage relationship in a graphical format or a table format.

2. The method according to claim 1, wherein The querying the data to be queried through a recursive algorithm includes: Querying the data to be queried in the specified database through a recursive algorithm.

3. The method according to claim 2, wherein After obtaining the data lineage query instruction input by the user, it further includes: Detecting the data to be queried; Determining that the data to be queried is located in the specified database according to the inspection result.

4. The method according to claim 2, wherein The data to be queried contains a target system name, a combination of a target system name and a target table name, or a combination of a target system name, a target table name, and a target field name.

5. The method according to any one of claims 1 to 4, characterized in that, After determining the data lineage relationship of the data to be queried based on the associated data and the dependency relationship, and displaying the data lineage relationship in a specified format, it further includes: Verifying the data lineage relationship; When it is determined that the data lineage relationship is in an abnormal state according to the verification result, an alarm prompt is given.

6. A data lineage acquisition device, characterized in that Including: A data query instruction acquisition module, configured to obtain a data lineage query instruction input by the user, where the data lineage query instruction contains the data to be queried; An associated data acquisition module, configured to query the data to be queried through a recursive algorithm to obtain associated data that has a dependency relationship with the data to be queried, where the dependency relationship includes a direct dependency relationship or an indirect dependency relationship; A lineage display module, configured to determine the data lineage of the data to be queried according to the associated data and the dependency relationship, and display the data lineage in a specified format, where the data lineage is used to identify the associated data and corresponding relationships involved from the generation to the invalidation of the data to be queried; The apparatus further includes: a specified database determination module, configured to obtain original data; store the original data in a specified database according to a preset path, where the preset path includes a path composed of a system name, a table name, and a field name; The associated data acquisition module is configured to: perform recursive hierarchical queries from the specified database according to the data to be queried to obtain first associated data that has a direct dependency relationship with the data to be queried; obtain second associated data that has an indirect dependency relationship with the data to be queried based on the first associated data; The lineage display module is configured to: determine a mapping rule according to the first associated data and the direct dependency relationship, and determine a filtering condition according to the second associated data and the indirect dependency relationship; determine the data lineage according to the first associated data, the second associated data, the mapping rule, and the filtering condition; display the data lineage in a graphical format or a table format.

7. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1-5.

Images