Data lineage determination method, device, equipment, storage medium and program product

Abstract

A data blood relationship determination method, device, equipment, storage medium and program product are disclosed. The present application relates to the technical field of databases. The method comprises: obtaining a data processing log, a job running record and a set mapping relationship; analyzing the data processing log to obtain a plurality of initial database operation statements; filtering the plurality of initial database operation statements based on the job running record and the set mapping relationship to obtain at least one target database operation statement; and determining a data blood relationship based on the at least one target database operation statement; wherein the data blood relationship comprises a table blood relationship and a field blood relationship. According to the scheme of the present application, the database operation statements are filtered based on the job running record and the set mapping relationship, and the data blood relationship is determined based on the filtered database operation statements, which can improve the efficiency and accuracy of data blood relationship determination.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and particularly relate to a data bloodline determination method, device, equipment, storage medium and program product. BACKGROUND

[0002] At present, when problems of financial data are investigated, due to lack of views of data processing relationship, only partial problems can be investigated, and cannot be corrected in time from the source. Therefore, it is particularly important to determine the data bloodline relationship. In the prior art, the data bloodline relationship is maintained by manual means, recorded by meta data management tools, or maintained by code scanning, which has the problems of low efficiency and inaccuracy. SUMMARY

[0003] Embodiments of the present application provide a data bloodline determination method, device, equipment, storage medium and program product, which can improve the efficiency and accuracy of data bloodline relationship determination.

[0004] In a first aspect, embodiments of the present application provide a data bloodline determination method, comprising:

[0005] obtaining data processing logs, job running records and a set mapping relationship; wherein the set mapping relationship is a mapping relationship between an Internet Protocol address and a host name;

[0006] parsing the data processing logs to obtain a plurality of initial database operation statements;

[0007] filtering the plurality of initial database operation statements based on the job running records and the set mapping relationship to obtain at least one target database operation statement;

[0008] determining a data bloodline relationship based on the at least one target database operation statement; wherein the data bloodline relationship includes a table bloodline relationship and a field bloodline relationship.

[0009] In a second aspect, embodiments of the present application also provide a data bloodline determination device, comprising:

[0010] an obtaining module configured to obtain data processing logs, job running records and a set mapping relationship; wherein the set mapping relationship is a mapping relationship between an Internet Protocol address and a host name;

[0011] an initial database operation statement obtaining module configured to parse the data processing logs to obtain a plurality of initial database operation statements;

[0012] a target database operation statement acquisition module, configured to filter the plurality of initial database operation statements based on the job running record and the setting mapping relationship, and obtain at least one target database operation statement;

[0013] a blood relationship determination module, configured to determine a data blood relationship based on the at least one target database operation statement; wherein the data blood relationship includes a table blood relationship and a field blood relationship.

[0014] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the data blood relationship determination method according to the embodiments of the present application when executing the program.

[0015] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program executable by a processor to implement the data blood relationship determination method according to the embodiments of the present application.

[0016] In a fifth aspect, a computer program product is provided, which includes a computer program executable by a processor to implement the data blood relationship determination method according to the embodiments of the present application.

[0017] The embodiments of the present application disclose a data blood relationship determination method, device, equipment, storage medium and program product, obtain a data processing log, a job running record and a setting mapping relationship; wherein the setting mapping relationship is a mapping relationship between an Internet Protocol address and a host name; analyze the data processing log to obtain a plurality of initial database operation statements; filter the plurality of initial database operation statements based on the job running record and the setting mapping relationship to obtain at least one target database operation statement; determine a data blood relationship based on the at least one target database operation statement; wherein the data blood relationship includes a table blood relationship and a field blood relationship. The data blood relationship determination method provided by the embodiments of the present application filters database operation statements based on a job running record and a setting mapping relationship to determine a data blood relationship based on the filtered database operation statements, which can improve the efficiency and accuracy of data blood relationship determination. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 A flow chart of a data blood relationship determination method provided for the embodiment one of the present application;

[0020] Figure 2 A structural schematic diagram of a data blood relationship determination device provided for the embodiment one of the present application;

[0021] Figure 3 A structural schematic diagram of an electronic device provided for the embodiment one of the present application. DETAILED DESCRIPTION

[0022] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0023] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second" and the like are only used to distinguish the description, but cannot be understood as indicating or implying relative importance. The acquisition, storage, use, processing and the like of data in the technical solution of the present application all comply with the relevant provisions of the national laws and regulations.

[0024] Figure 1 A flow chart of a data blood relationship determination method provided for the embodiment of the present application, the method is suitable for determining the data blood relationship, the method can be executed by a data blood relationship determination device, the device can be realized in the form of software and / or hardware, and optionally, realized by an electronic device, which can be a mobile terminal, a PC terminal or a server, etc. As shown in the figure, the method specifically includes the following steps: Figure 1

[0025] S110, acquiring data processing logs, job running records and a set mapping relationship.

[0026] The set mapping relationship is the mapping relationship between the Internet Protocol Address (IP) and the host name. The data processing logs can be the logs generated by the extract-transform-load (ETL) script running on the database. The job running records can be the running records corresponding to the data processing business in a certain business scenario.

[0027] ​In the embodiment, the data processing log comprises a plurality of data processing records, each data processing record comprises information of an Internet Protocol address and a job start time. The job running record comprises information of a job running period and a host name. The job running period can be composed of the job start time and a job end time.

[0028] S120, parsing the data processing log to obtain a plurality of initial database operation statements.

[0029] In the embodiment, the data processing log can be parsed in the following manner: the existing log reader is used to read and parse the data processing log, so as to obtain the database operation statements (Structured Query Language, SQL) corresponding to each data processing record in the data processing log, i.e. a plurality of initial SQL statements. The log reader can be a CSV reader, which is not limited herein. The database operation statement can comprise a data addition statement, a data deletion statement, a data modification statement and a data query statement.

[0030] S130, screening the plurality of initial database operation statements based on the job running record and the set mapping relationship to obtain at least one target database operation statement.

[0031] Each initial SQL statement has its corresponding IP address and job start time. The plurality of initial database operation statements can be screened based on the job running record and the set mapping relationship in the following manner: the job running period and the host name in the job running record are extracted, the IP address corresponding to the host name is determined according to the set mapping relationship, and finally the SQL statement corresponding to the IP address and falling within the job running period is screened out from the initial SQL statement.

[0032] Specifically, the plurality of initial database operation statements can be screened based on the job running record and the set mapping relationship to obtain at least one target database operation statement in the following manner: the host name corresponding to the Internet Protocol address in the data processing log is determined according to the set mapping relationship; the job running period corresponding to the host name is determined according to the job running record; the initial database operation statement in the data processing log falling within the job running period is screened out, and at least one target database operation statement is obtained.

[0033] In this embodiment, it is assumed that the Internet protocol address in the data processing log is IP1, the host name corresponding to IP1 in the setting mapping relationship is A, then the job running period T1-T2 when the job is executed on the host A is determined according to the job running record, and finally the initial SQL statement in the data processing log whose job start time t falls into T1-T2 is screened out to obtain at least one target SQL statement. In this embodiment, the SQL statement is screened based on the IP address and the job start time, so as to ensure that the extracted SQL statement corresponds to the same data life line, thereby ensuring the accuracy of the subsequently determined data blood relationship.

[0034] In S140, a data blood relationship is determined based on the at least one target database operation statement.

[0035] The data blood relationship includes a table blood relationship and a field blood relationship. The table blood relationship can be understood as the tables through which the data passes in the entire processing life cycle, and the processing order of each table. The field blood relationship can be understood as the fields corresponding to the data in the entire processing life cycle, and the processing order of each field.

[0036] Optionally, the manner of determining the data blood relationship based on the at least one target database operation statement can be: analyzing the at least one target database operation statement to obtain the source table and target table relationship of each target database operation statement, and / or the source field and target field relationship; constructing the table blood relationship based on the source table and target table relationship of each target database operation statement, and constructing the field blood relationship based on the source field and target field relationship of each target database operation data.

[0037] The source table can be understood as the table from which the data flows when the data is operated according to the SQL statement, and the target table can be understood as the table into which the data flows when the data is operated according to the SQL statement. The source field can be understood as the field corresponding to the data in the source table or the field corresponding to the data before processing, and the target field can be understood as the field corresponding to the data in the target table or the field corresponding to the data after processing. The source table and target table relationship can be represented as: source table-target table; and the source field and target field relationship can be represented as: source field-target field.

[0038] A certain table is a source table in one SQL statement, and can be a target table in another SQL statement. Similarly, a certain field is a source field in one SQL statement, and can be a target field in another SQL statement. The source table corresponding to one SQL statement is one or more, and the target table corresponding to one SQL statement is also one or more. The source field corresponding to one SQL statement is one or more, and the target field corresponding to one SQL statement is also one or more.

[0039] When a certain SQL statement is executed, data can be transferred between tables or only changed within a table. When data is transferred between tables, the source table and target table relationship corresponding to the SQL statement can be obtained, and the source field and target field relationship can be obtained. When data is changed within a table, the source field and target field relationship corresponding to the SQL statement can be obtained.

[0040] Specifically, the manner of constructing the table blood relationship based on the source table and target table relationship of each target database operation statement can be: concatenating each source table and target table relationship to obtain the table blood relationship. The manner of constructing the field blood relationship based on the source field and target field relationship of each target database operation data can be: concatenating the source field and target field relationship to obtain the field blood relationship.

[0041] Optionally, the process of constructing the table blood relationship based on the source table and target table relationship of each target database operation statement and constructing the field blood relationship based on the source field and target field relationship of each target database operation data can be: determining an end table; filtering the source table and target table relationship and the source field and target field relationship based on the end table; constructing the table blood relationship based on the filtered source table and target table relationship, and constructing the field blood relationship based on the filtered source field and target field relationship.

[0042] The end table can be the last table flowing into the data processing cycle, and the end table can include one or more. Filtering the source table and target table relationship and the source field and target field relationship based on the end table can be understood as: filtering out the source table and target table relationship irrelevant to the end table and the source field and target field relationship irrelevant to the end table. In this embodiment, filtering the source table and target table relationship and the source field and target field relationship based on the end table can improve the accuracy of the data blood relationship.

[0043] Optionally, the manner of filtering the source table and target table relationship and the source field and target field relationship based on the end table can be: starting from the end table, tracing back to the start table based on the source table and target table relationship to obtain multiple table links; starting from the field corresponding to the end table, tracing back to the start field based on the source field and target field relationship to obtain multiple field links; determining the source table and target table relationship not in any table link as irrelevant source table and target table relationship; determining the source field and target field relationship not in any field link as irrelevant source field and target field relationship; and filtering out the irrelevant source table and target table relationship and the irrelevant source field and target field relationship.

[0044] The start field can be a field corresponding to the start table in the data. The source table and target table relationship unrelated to the end table can be understood as a source table and target table relationship unrelated to the end table. The source field and target field relationship unrelated to the end table can be understood as a source field and target field relationship unrelated to the end table. The process of tracing back from the end table to the start table based on the source table and target table relationship can be: first, determining the source table corresponding to the end table as the target table based on the source table and target table relationship, then determining the source table corresponding to the end table as the target table, and so on, until the start table is reached. Thus, a plurality of table links corresponding to the end table are obtained. The process of tracing back from the field corresponding to the end table to the start field based on the source field and target field relationship can be: first, determining the field corresponding to the end table (also referred to as the end field) as the source field corresponding to the target field based on the source field and target field relationship, then determining the source field corresponding to the end table as the target field, and so on, until the start field is reached. Thus, a plurality of field links corresponding to the end field are obtained.

[0045] After obtaining the plurality of table links and the plurality of field links, the source table and target table relationship unrelated to any table link is determined as the source table and target table relationship unrelated to the end table, and the source field and target field relationship unrelated to any field link is determined as the source field and target field relationship unrelated to the end table. Finally, the source table and target table relationship unrelated to the end table and the source field and target field relationship unrelated to the end table are filtered out.

[0046] Specifically, the manner of constructing the table blood relationship based on the filtered source table and target table relationship and constructing the field blood relationship based on the filtered source field and target field relationship can be: connecting the tables based on the filtered source table and target table relationship to obtain a table connection network; and taking the table connection network as the table blood relationship.

[0047] Specifically, the process of connecting the tables based on the filtered source table and target table relationship can be: starting from the start table, sequentially connecting the tables based on the filtered source table and target table relationship until the end table is reached.

[0048] Specifically, the manner of constructing the field blood relationship based on the filtered source field and target field relationship can be: connecting the fields based on the filtered source field and target field relationship to obtain a field connection network; and taking the field connection network as the field blood relationship.

[0049] Specifically, the process of connecting the fields based on the filtered source field and target field relationship can be: starting from the start field, sequentially connecting the fields based on the filtered source field and target field relationship until the end field is reached.

[0050] Optionally, after determining the data blood relationship based on the at least one target database operation statement, the method further comprises: dividing the plurality of data blood relationships into business scenarios based on the job running time period in the job running record.

[0051] The job running time period has a corresponding relationship with the business scenario, so the plurality of data blood relationships can be divided into business scenarios based on the job running time period in the job running record, and the business scenario corresponding to each data blood relationship is obtained. This facilitates subsequent query of the data blood relationship based on the business scenario.

[0052] Optionally, the method further comprises: querying the data blood relationship based on the business scenario and the end table, and displaying the queried data blood relationship.

[0053] Specifically, when a user wants to query the data blood relationship of a certain business scenario, the business scenario information and the end table information are input, the system queries the corresponding data blood relationship based on the business scenario and the end table, and displays the queried data blood relationship. In this embodiment, the table level relationship and the field corresponding relationship between tables can be displayed from the end table to the starting table in sequence.

[0054] The technical scheme of this embodiment acquires a data processing log, a job running record and a set mapping relationship, wherein the set mapping relationship is a mapping relationship between an Internet protocol address and a host name; analyzes the data processing log to obtain a plurality of initial database operation statements; filters the plurality of initial database operation statements based on the job running record and the set mapping relationship to obtain at least one target database operation statement; determines a data blood relationship based on the at least one target database operation statement; and wherein the data blood relationship comprises a table blood relationship and a field blood relationship. The data blood relationship determination method provided in this embodiment filters the database operation statements based on the job running record and the set mapping relationship to determine the data blood relationship based on the filtered database operation statements, which can improve the efficiency and accuracy of data blood relationship determination.

[0055] Figure 2 is a structural schematic diagram of a data blood relationship determination device provided in this embodiment, as shown in Figure 2 The device comprises:

[0056] The acquisition module 210 is configured to acquire a data processing log, a job running record and a set mapping relationship, wherein the set mapping relationship is a mapping relationship between an Internet protocol address and a host name.

[0057] The initial database operation statement acquisition module 220 is configured to analyze the data processing log to obtain a plurality of initial database operation statements.

[0058] The target database operation statement obtaining module 230 is configured to filter the plurality of initial database operation statements based on the job running record and the set mapping relationship, and obtain at least one target database operation statement.

[0059] The blood relationship determining module 240 is configured to determine a data blood relationship based on the at least one target database operation statement; wherein the data blood relationship comprises a table blood relationship and a field blood relationship.

[0060] Optionally, the data processing log comprises an Internet Protocol address and a job starting time; and the job running record comprises a job running period and a host name.

[0061] Optionally, the target database operation statement obtaining module 230 is further configured to:

[0062] determine the host name corresponding to the Internet Protocol address in the data processing log according to the set mapping relationship;

[0063] determine the job running period corresponding to the host name according to the job running record;

[0064] filter out the initial database operation statement with the job starting time falling within the job running period in the data processing log, and obtain at least one target database operation statement.

[0065] Optionally, the blood relationship determining module 240 is further configured to:

[0066] analyze the at least one target database operation statement, and obtain the source table and target table relationship of each target database operation statement, and / or the source field and target field relationship;

[0067] construct the table blood relationship based on the source table and target table relationship of each target database operation statement, and construct the field blood relationship based on the source field and target field relationship of each target database operation data.

[0068] Optionally, the blood relationship determining module 240 is further configured to:

[0069] determine an end table;

[0070] filter the source table and target table relationship and the source field and target field relationship based on the end table;

[0071] construct the table blood relationship based on the filtered source table and target table relationship, and construct the field blood relationship based on the filtered source field and target field relationship.

[0072] Optionally, the blood relationship determining module 240 is further configured to:

[0073] obtaining a plurality of table links from the end table to a start table based on the source table-target table relationship;

[0074] obtaining a plurality of field links from the end table corresponding field to a start field based on the source field-target field relationship;

[0075] determining a source table-target table relationship not in any table link as an irrelevant source table-target table relationship;

[0076] determining a source field-target field relationship not in any field link as an irrelevant source field-target field relationship;

[0077] filtering out the irrelevant source table-target table relationship and the irrelevant source field-target field relationship.

[0078] Optionally, the blood relationship determining module 240 is further configured to:

[0079] connecting tables based on the filtered source table-target table relationship to obtain a table connection network;

[0080] taking the table connection network as a table blood relationship.

[0081] Optionally, the blood relationship determining module 240 is further configured to:

[0082] connecting fields based on the filtered source field-target field relationship to obtain a field connection network;

[0083] taking the field connection network as a field blood relationship.

[0084] Optionally, the apparatus further comprises a business scenario dividing module configured to

[0085] dividing a plurality of data blood relationships based on an action running period in the job running record.

[0086] Optionally, the apparatus further comprises a display module configured to display a queried data blood relationship based on the business scenario and the end table query data blood relationship.

[0087] The apparatus can execute the method provided by all the foregoing embodiments of the application, and has corresponding function modules and beneficial effects for executing the foregoing method. Technical details not described in the embodiment can be found in the method provided by all the foregoing embodiments of the application.

[0088] Figure 3A structural diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0089] As shown, Figure 3 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., connected in communication with the at least one processor 11, where the memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer programs stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0090] Various components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0091] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the data lineage determination method.

[0092] In some embodiments, the method of determining data lineage can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 18. In some embodiments, portions or all of the computer program can be loaded and / or installed onto electronic device 10 via, e.g., ROM 12 and / or communication unit 19. When the computer program is loaded onto RAM 13 and executed by processor 11, one or more steps of the method of determining data lineage described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method of determining data lineage by other means, e.g., with the aid of firmware.

[0093] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0094] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program, when executed by the processor of the machine, implements the functions / acts specified in the flowcharts and / or block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0095] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0096] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0097] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0098] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0099] The embodiment of the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the data blood relationship determination method provided in any embodiment of the present application.

[0100] The computer program product can be written in one or more programming languages or combinations of languages including object-oriented languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0101] Note that the above only describes the preferred embodiments of the present application and the applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A method for determining data lineage, characterized in that, include: Acquire data processing logs, job execution records, and set mapping relationships; wherein, the set mapping relationship is the mapping relationship between Internet Protocol addresses and hostnames; The data processing log is parsed to obtain multiple initial database operation statements; Based on the job execution records and the defined mapping relationship, the multiple initial database operation statements are filtered to obtain at least one target database operation statement; The data lineage is determined based on the at least one target database operation statement; wherein, the data lineage includes table lineage and field lineage; The data processing log includes: Internet Protocol address and job start time; the job execution record includes: job execution period and hostname. Based on the job execution records and the defined mapping relationship, the multiple initial database operation statements are filtered to obtain at least one target database operation statement, including: The hostname corresponding to the Internet Protocol address in the data processing log is determined according to the established mapping relationship. Determine the job runtime segment corresponding to the hostname based on the job execution record; The initial database operation statements in the data processing log whose job start time falls within the job's runtime segment are filtered out to obtain at least one target database operation statement.

2. The method according to claim 1, characterized in that, Determining data lineage based on at least one target database operation statement includes: The at least one target database operation statement is parsed to obtain the relationship between the source table and the target table for each target database operation statement, and / or the relationship between the source field and the target field. Table lineage is constructed based on the relationship between the source table and the target table in the operation statements of each target database, and field lineage is constructed based on the relationship between the source field and the target field in the operation data of each target database.

3. The method according to claim 2, characterized in that, Table lineage is constructed based on the relationship between the source table and target table in each target database operation statement, and field lineage is constructed based on the relationship between the source field and target field in each target database operation data, including: Determine the end table; Filtering is performed based on the relationship between the source table and the target table, and the relationship between the source fields and the target fields, using the terminal table as the basis; Table lineage is constructed based on the filtered relationship between the source table and the target table, and field lineage is constructed based on the filtered relationship between the source fields and the target fields.

4. The method according to claim 3, characterized in that, Filtering based on the relationship between the source table and the target table, and the relationship between the source fields and the target fields, using the terminal table, includes: Starting from the end table, trace back to the starting table based on the relationship between the source table and the target table to obtain multiple table links; Starting from the field corresponding to the end table, trace back to the starting field based on the relationship between the source field and the target field to obtain multiple field links; The relationship between a source table and a target table that is not in any table link is defined as an unrelated source table and target table relationship. The relationship between a source field and a target field that is not in any field link is defined as an unrelated source field and target field relationship; The relationships between the unrelated source table and the target table, as well as the relationships between the unrelated source fields and the target fields, are filtered out.

5. The method according to claim 3, characterized in that, Based on the filtered relationship between the source and target tables, table lineage is constructed; based on the filtered relationship between source and target fields, field lineage is constructed, including: The tables are joined based on the filtered relationship between the source and target tables to obtain a table join network; The table connection network is used as the table lineage.

6. The method according to claim 3, characterized in that, Based on the relationship between the filtered source field and the target field, a field lineage relationship is constructed, including: Based on the relationship between the filtered source and target fields, the fields are connected to obtain a field connection network; Connect the fields to the network to determine the field lineage.

7. The method according to claim 3, characterized in that, After determining the data lineage based on the at least one target database operation statement, the method further includes: Based on the operational runtime segments in the job execution records, business scenarios are divided according to multiple data lineages.

8. The method according to claim 7, characterized in that, Also includes: Based on the aforementioned business scenario and the query data lineage relationship in the terminal table, the queried data lineage relationship is displayed.

9. A device for determining data lineage, characterized in that, include: The acquisition module is used to acquire data processing logs, job execution records, and set mapping relationships; wherein, the set mapping relationship is the mapping relationship between Internet Protocol addresses and hostnames; The initial database operation statement acquisition module is used to parse the data processing log to obtain multiple initial database operation statements; The target database operation statement acquisition module is used to filter the multiple initial database operation statements based on the job execution record and the set mapping relationship to obtain at least one target database operation statement. A lineage determination module is used to determine data lineage based on the at least one target database operation statement; wherein, the data lineage includes table lineage and field lineage; The data processing log includes: Internet Protocol address and job start time; the job execution record includes: job execution period and hostname. The target database operation statement acquisition module is also used for: The hostname corresponding to the Internet Protocol address in the data processing log is determined according to the established mapping relationship. Determine the job runtime segment corresponding to the hostname based on the job execution record; The initial database operation statements in the data processing log whose job start time falls within the job's runtime segment are filtered out to obtain at least one target database operation statement.

10. An electronic device, characterized in that, The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the method for determining data lineage as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method for determining data lineage as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for determining data lineage as described in any one of claims 1-8.

Images