Data blood relationship construction method and system realized based on input and output URN, and storage medium

Abstract

The embodiment of the invention provides a data consanguinity construction method and system based on input and output URN and a storage medium. The method comprises the steps that a task needing to be processed is received, MapTask is started and executed for the task, and input data needed by the task are sequentially read; assembling a uniform URN according to a preset input and output flow URN rule and data information of the input data, and after the task executes the MapTask, analyzing according to an input flow URN and an output flow URN of the MapTask to obtain a blood relationship two-tuple of a source subsystem, a source database, a source data table, a target subsystem, a target database, a target data table and acquisition time; and writing the blood relationship two-tuple into a blood relationship database to realize blood relationship construction.

Description

Technical Field

[0001] The present application belongs to the field of computer software and big data technology, and specifically relates to a data lineage construction method, system and storage medium based on input and output URN. Background Art

[0002] The Data Lineage system in the Qitong Big Data system tracks and records the flow and transformation of data within business processes, helping us understand how data flows and transforms across different environments and applications. It reveals the data's history, transmission, and changes, ensuring data quality, compliance, and credibility. The Data Lineage system primarily improves data governance efficiency, optimizes data asset management, enhances data security, improves the experience of data products and applications, and facilitates verification of data processing logic. Its importance in big data security is evident in this system.

[0003] The existing technology uses an SQL parser to statically parse batch scripts implemented in the SQL language, which can obtain the relationship between the source data table and the target data table in the batch script, and then build a data lineage map based on the relationship between the tables. The existing technology can only solve batch scripts in the SQL language, and cannot parse and obtain lineage relationships for batch scripts in other types of languages ​​(such as shell, python, java and other assembly languages). Summary of the Invention

[0004] This application provides a data lineage construction method, system and storage medium based on input and output URN, which can realize the establishment of lineage relationships in other types of languages.

[0005] In a first aspect, the present application provides a data lineage construction method based on input and output URNs, the method comprising the following steps:

[0006] Receive the task to be processed, start executing MapTask for the task, and read the input data required by the task in sequence;

[0007] According to the preset input and output stream URN rules and the data information of the input data, a unified URN is assembled. When the task completes the MapTask, the input stream URN and output stream URN of the MapTask are parsed to obtain the blood relationship tuple of the source subsystem, source database, source data table, target subsystem, target database, target data table, and collection time;

[0008] The blood relationship tuple is written into the blood relationship database to realize the construction of blood relationship.

[0009] In a second aspect, the present application provides a data lineage construction system based on input and output URNs, the system comprising:

[0010] A receiving unit is used to receive a task to be processed, start executing MapTask for the task, and sequentially read input data required for the task;

[0011] The analysis unit is used to assemble a unified URN according to the preset input and output stream URN rules and the data information of the input data. When the task completes the MapTask, it parses the input stream URN and output stream URN of the MapTask to obtain the blood relationship tuple of the source subsystem, source database, source data table, target subsystem, target database, target data table, and collection time;

[0012] The construction unit is used to write the blood relationship tuple into the blood relationship database to realize the construction of blood relationship.

[0013] In a third aspect, the present application provides a computer storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute some or all of the steps described in the first aspect of the present application.

[0014] The embodiments of the present application have the following beneficial effects:

[0015] The technical solution of this application constructs a data lineage system through the URN of data input and output streams, solves the relationship of data flow from the perspective of underlying data flow, and fundamentally solves the problem of unresolvable lineage relationships caused by different upper-level non-SQL applications. It does not rely on manual maintenance of lineage relationships, and the constructed lineage relationships are more accurate. In addition, lineage relationships can be traced back based on different time snapshots. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 It is a structural diagram of the data center provided by this application;

[0018] Figure 2 This is a flowchart of a data lineage construction method based on input and output URNs provided by this application;

[0019] Figure 3 A schematic diagram of a user verification process provided for this application;

[0020] Figure 4A schematic diagram of a blood relationship dyad adopted for this application;

[0021] Figure 5 A flowchart of a data lineage construction method based on input and output URNs provided in Example 1 of the present application;

[0022] Figure 6 This is a structural diagram of a data lineage construction system based on input and output URNs provided in this application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, system, product, or apparatus.

[0025] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0026] The following is an introduction to the relevant terms involved in this application.

[0027] Data lineage: Data lineage is a crucial concept that describes the relationships between data. It refers to the complete record of the transformations and processing that data undergoes from its original source to its final destination. Data lineage not only describes the entire lifecycle of data, including its generation, processing, processing, integration, circulation, and ultimate disappearance, but also records the links between data during these processes. These relationships are similar to human blood ties, hence the name data lineage.

[0028] Sqoop: Sqoop is an open source tool that is primarily used for efficient data transfer between Hadoop (Hive) and traditional databases such as MySQL and PostgreSQL. It can import data from relational databases into Hadoop's HDFS and export data from HDFS to relational databases.

[0029] Spark: Apache Spark is a fast and general-purpose big data processing framework with in-memory computing capabilities. It can be hundreds of times faster than Hadoop MapReduce, supports interactive computing and complex algorithms, and provides a variety of function libraries including SQL queries, text processing, and machine learning.

[0030] Spark RDD: Spark RDD (Resilient Distributed Dataset) is the core data abstraction in Spark. It represents an immutable, partitioned, parallelizable distributed dataset with elastic storage, fault tolerance, and distributed computing features. It is the cornerstone of Spark's big data processing.

[0031] MapReduce: MapReduce is a programming model for parallel computing of large-scale data sets (greater than 1TB). It abstracts the complex parallel computing process running on large-scale clusters into two functions: Map and Reduce.

[0032] Flink: Flink is a high-performance, high-throughput open source stream processing and batch processing framework that supports real-time data processing and large-scale data set computing. It has the characteristics of fault tolerance, scalability, and low latency, and is suitable for stateful computing of unbounded and bounded data streams.

[0033] URN: A Uniform Resource Name (URN) is a naming scheme used to uniquely identify and reference resources on the internet. It is persistent and unique, independent of the resource's location or access method, making it suitable for information exchange, document management, and persistent storage. URNs assign unique identifiers to different types of resources using a specific namespace and naming scheme, such as UUIDs and ISBNs.

[0034] Hadoop: Hadoop is a distributed system infrastructure that is primarily used to store, process, and analyze large data sets on a cluster. It stores data through the Hadoop Distributed File System (HDFS) and processes data through programming models such as MapReduce.

[0035] Kerberos: Kerberos is a computer network authentication protocol developed at MIT. It is designed to provide strong client / server authentication services over insecure networks using a key system. Designed based on a trusted third-party authentication service, it supports single sign-on and is widely used in web applications, remote login, and wireless networks to ensure secure communications.

[0036] RPC: RPC (Remote Procedure Call) is a protocol that requests services from remote computer programs over a network without having to understand the underlying network technology. It allows calling services on remote computers to be as simple as calling local services.

[0037] The following describes the system architecture of a data center related to an embodiment of the present application.

[0038] The present application also provides a data center. The data lineage of the present application can be Figure 1 The data center shown is established in Figure 1 The above data center includes: user authority management center, job management center, URN rule configuration center, URN collector, URN analysis and processing, lineage construction module, ResourceManager, NodeManager, ApplicationMaster, MapTask and ReduceTask, etc.

[0039] ResourceManager: A key component in distributed systems such as Hadoop YARN and Flink, ResourceManager is responsible for the unified management and allocation of cluster resources. It receives node resource feedback information and allocates it to applications based on policies, ensuring efficient resource utilization and smooth job execution.

[0040] NodeManager: NodeManager is a key component in Hadoop YARN. It runs on a single node and is responsible for managing the resources of a single computing node in the Hadoop cluster. This includes communicating with the ResourceManager, managing the Container lifecycle, monitoring Container resource usage, tracking node health, managing logs and ancillary services, etc., to ensure efficient cluster operation and resource optimization.

[0041] ApplicationMaster: The ApplicationMaster is a core component in distributed computing frameworks such as Hadoop YARN. It coordinates and manages resource requests, task allocation, execution monitoring, and failure handling for specific applications. It is a framework-initiated component that runs independently on the YARN cluster. By interacting with the ResourceManager and NodeManager, it ensures efficient and reliable application execution within the cluster.

[0042] MapTask: A MapTask is the fundamental computational unit in distributed computing frameworks such as Hadoop and Spark. It is responsible for reading input data, converting it into key-value pairs, applying a map function to it, and outputting intermediate results. In Hadoop, a MapTask reads data from an InputSplit using a RecordReader, then applies a user-defined map function and stores the results in a circular memory buffer. When the buffer is full, the data is written to a temporary file on the local disk and ultimately merged into a single data file. In Spark, a MapTask is responsible for segmenting, transforming, and computing the initial input data, generating intermediate results for use by a ReduceTask.

[0043] The specific methods are introduced in detail below.

[0044] See also Figure 2 This application also provides a data lineage construction method based on input and output URN. Figure 2 This is a flow chart of a data lineage construction method based on input and output URNs provided by this application. The above method can be used in Figure 1 The above method is executed on the data center shown in FIG. Figure 2 As shown, the following steps are included:

[0045] Step S201: Receive a task to be processed, start executing MapTask for the task, and read the input data required by the task in sequence;

[0046] See Figure 3 , Figure 3 This is a user verification process diagram provided by this application. The above method is performed before step S201. Figure 3 As shown, the above user verification process includes the following steps:

[0047] Step S301: User A submits a job to the cluster using the hadoop command.

[0048] Step S302: The Hadoop client queries the user authority management center to see whether user A who submitted the job has sufficient authority.

[0049] Step 303: If the user uses Kerberos authentication when submitting the job, the Hadoop client will verify the user's Kerberos credentials.

[0050] Step 304: After the user is authenticated, the Hadoop client submits the job to the ResourceManager by calling the RPC interface.

[0051] Step 305: After receiving the job request, ResourceManager queries the user authority management center whether the user who submitted the job has the authority to execute the job. This usually involves whether the user belongs to a specific user group or has a specific role or authority.

[0052] Step 306: ResourceManager allocates the task to the corresponding NodeManager for execution through ApplicationMaster based on the NodeManager's resource situation and job requirements.

[0053] Step 307: NodeManager and ResourceManager maintain communication through heartbeats.

[0054] In an optional technical scenario, the input data can be batch tasks written in multiple languages, such as SQL, Shell, Python, and Java. Specifically, these batch tasks include, but are not limited to, Sqoop data extraction tasks written in Shell, Spark data processing tasks implemented in Python, MapReduce computing tasks implemented in Java, and Flink real-time computing tasks.

[0055] In an optional technical scenario, the above method may further include, after step S201:

[0056] Get the input data storage capacity. If the storage capacity exceeds the storage threshold, overflow the historical data in the memory and transfer it to the file system. This historical data will be used as the intermediate result cache of the job.

[0057] This method can process input data with a large storage capacity, avoid the problem of insufficient remaining memory to process the input data, and improve the stability of the system.

[0058] Step S202: Assemble a unified URN based on the preset input and output stream URN rules and the data information of the input data. After the task completes the MapTask, parse the input stream URN and output stream URN of the MapTask to obtain the blood relationship tuple of the source subsystem, source database, source data table, target subsystem, target database, target data table, and collection time.

[0059] In an optional technical scenario, the data information of the input data may specifically include: one or any combination of the IP address, port number, library name, table name, time, file path, and language type of the input stream data.

[0060] In an optional technical scenario, the above-mentioned assembly of the unified URN according to the preset input and output stream URN rules and the data information of the input data may specifically include:

[0061] The data information of the input data is matched with the preset input and output stream URN rules to obtain a rule ID number with the same information as the data information of the input data, and the data information of the input data is assembled into a unified URN according to the arrangement order of the rule expression of the rule ID number.

[0062] In an optional application scenario, assembling the data information of the input data into a unified URN according to the arrangement order of the rule expression of the rule ID number may specifically include:

[0063] Obtain the file path, data protocol, and database ID in the data information of the input data, and encapsulate the file path, data protocol, database ID, and table name into a unified URN according to the assembly order of the first URN rule corresponding to the rule ID number.

[0064] The above-mentioned file path includes but is not limited to: a combination of an IP address and a port number. Of course, it can also be other file paths. This application does not limit the specific form of the above-mentioned file path, for example, D:\E\lincoa\WeBank\2024-****-CN.

[0065] The following is an example to illustrate, assuming that the preset input and output flow URN rules are as shown in Table 1.

[0066] Table 1 URN rule table

[0067] ID Regular Expressions Cluster Subsystem Library Name Table name R001 jdbc:mysql: / / 10.10.1.20:11516 / $db / $table TDC1 S001 $db $table R002 jdbc:mysql: / / 10.10.1.22:11516 / $db / $table TDC1 S001 $db $table R003 jdbc:mysql: / / 10.10.1.25:11516 / $db / $table TDC1 S001 $db $table R004 jdbc:oracle:thin:@10.20.2.2:11011 / $db / $table ORC2 S002 $db $table R005 jdbc:oracle:thin:@10.20.2.8:11011 / $db / $table ORC2 S002 $db $table R006 hdfs: / / 10.10.3.10 / $db / $table HDC1 S003 $db $table R007 hdfs: / / 10.10.3.10 / tmp / $hash HDC1 S003

[0068] In the above rule expression, $table represents any table name, $db represents any database name, tmp represents a temporary directory identifier, $hash represents any hash value, hdfs represents the language type, jdbc:oracle:thin represents the database JDBC connection protocol identifier, and jdbc:mysql represents another database JDBC connection protocol identifier.

[0069] In an optional technical scenario, the data information of the above input data may include: IP address: 10.10.1.20, port number: 11516, language type (also known as protocol): jdbc, library name: database_a, table name: table_1, then the matched rule ID number is R001 (the protocol number, port number, and IP address of the above R001 are the same as the data information of the input data), then according to the above rule expression, the unified URN assembled can be: jdbc:mysql: / / 10.10.1.20:11516 / database_a / table_1 / .

[0070] In an optional technical scenario, the above-mentioned tuples of the lineage relationship of the source subsystem, source database, source data table, target subsystem, target database, target data table, and collection time obtained by parsing the input stream URN and output stream URN of MapTask may specifically include:

[0071] According to the input stream URN of MapTask, match the output stream URN corresponding to the input stream URN, obtain the target database, target data table and target subsystem corresponding to the output stream URN, obtain the source subsystem, source database and source data table corresponding to the input stream URN, construct the system tuple from the source subsystem to the target subsystem, and construct the blood relationship tuple from the source database and source data table to the target database and target data table.

[0072] The source subsystem, source database, and source data table corresponding to the input stream URN obtained above may specifically include:

[0073] Extract the first rule ID number matched by the input stream URN, extract the source subsystem corresponding to the first rule ID number in the preset input and output stream URN rules, and extract the source database and source data table from the input stream URN.

[0074] The target database, target data table, and target subsystem corresponding to the output stream URN obtained above may specifically include:

[0075] Extract the second rule ID number matched by the output stream URN, extract the subsystem corresponding to the second rule ID number from the preset input and output stream URN rules as the target subsystem, and extract the target database and target data table from the output stream URN.

[0076] The following example illustrates how to construct the corresponding tuple.

[0077] Refer to Table 2, which is the URN collection table.

[0078] Table 2:

[0079] ID Data Flow URN Assignment Collection time 1001 enter jdbc:mysql: / / 10.10.1.20:11516 / database_a / table_1 / J001 20240101 1002 enter jdbc:mysql: / / 10.10.1.25:11516 / database_a / table_2 / J001 20240101 1003 enter jdbc:oracle:thin:@10.20.2.2:11011 / database_b / table_3 / J001 20240101 1004 Output hdfs: / / 10.10.3.10 / tmp / 34d29f0ef03aedddd8ca45ba90fd89e2 J001 20240101 1005 Output hdfs: / / 10.10.3.10 / tmp / 4a787801be07547f8fdc2a305b49d2f7 J001 20240101 1006 Output hdfs: / / 10.10.3.10 / tmp / a8e22c9b8d11e65d6577cd0aeeba39b2 J001 20240101 1007 enter hdfs: / / 10.10.3.10 / tmp / 34d29f0ef03aedddd8ca45ba90fd89e2 J001 20240101 1008 enter hdfs: / / 10.10.3.10 / tmp / 4a787801be07547f8fdc2a305b49d2f7 J001 20240101 1009 enter hdfs: / / 10.10.3.10 / tmp / a8e22c9b8d11e65d6577cd0aeeba39b2 J001 20240101 1010 Output hdfs: / / 10.10.3.10 / database_c / table_4 / J001 20240101

[0080] Parsing Table 2, we determine that URNs with IDs 1001, 1002, 1003, and 1010 in Table 2 match URN rules with IDs R001, R003, R004, and R006 in Table 1, respectively. This analysis yields three pairs of bigrams, as shown in Table 3. The remaining six URNs (1004, 1005, ..., 1009) match R007. However, because the same URN exists in both input and output data flows, these are considered intermediate results and are ignored during analysis.

[0081] Table 3 Newly added tuple table

[0082] Source subsystem Source database Source data table Target subsystem Target database Target data table Collection time S001 database_a table_1 S003 database_c table_4 20240101 S001 database_a table_2 S003 database_c table_4 20240101 S002 database_b table_3 S003 database_c table_4 20240101

[0083] In this way, it is necessary to construct the blood relationship tuples of database_a table_1 to database_c table_4, database_atable_2 to database_ctable_4, and database_a table_5 to database_c table_4. Figure 4 , Figure 4 This is a schematic diagram of a blood relationship binary group approved by this application.

[0084] Step S203: Write the blood relationship tuple into the blood relationship database to realize the construction of blood relationship.

[0085] The technical solution of this application constructs a data lineage system through the URN of data input and output streams, solves the relationship of data flow from the perspective of underlying data flow, and fundamentally solves the problem of unresolvable lineage relationships caused by different upper-level non-SQL applications. It does not rely on manual maintenance of lineage relationships, and the constructed lineage relationships are more accurate. In addition, lineage relationships can be traced back based on different time snapshots.

[0086] In an optional application scenario, the above-mentioned writing of the blood relationship tuple into the blood relationship database to realize blood relationship construction may specifically include:

[0087] If the blood relationship data already exists in the database, the collection time will be updated; if the blood relationship does not exist, the blood relationship will be inserted into the blood relationship database.

[0088] In an optional application scenario, inserting the blood relationship into the blood relationship database may specifically include:

[0089] The source database and source data table are used as the starting point of the data lineage graph, the target database and target data table are used as the end point of the data lineage graph, the association relationship is used as the edge of the data lineage graph, and the collection time is used as the attribute on the directed edge to construct a data lineage graph from the starting point to the end point.

[0090] Example 1

[0091] This application also provides a data lineage construction method based on input and output URN. Figure 5 This is a flow chart of a data lineage construction method based on input and output URNs provided by this application. The above method can be used in Figure 1 The URN rule table of the embodiment of the present application can be shown in Table 1 above, which will not be repeated here. The URN collection table of the embodiment of the present application can be shown in Table 2, which will not be repeated here.

[0092] The above method is as follows Figure 5 As shown, the following steps are included:

[0093] Step S501: Receive a task to be processed, start executing MapTask for the task, and sequentially read the input data required for the task;

[0094] Step S502: Analyze and process the collected URNs according to the URN rules, obtain the association relationship of the data, and obtain the edge set of the binary.

[0095] Specifically, the URNs with IDs 1001, 1002, 1003, and 1010 in Table 2 hit the URN rules with IDs R001, R003, R004, and R006 in Table 1, respectively. Three pairs of tuples are obtained from the analysis, as shown in Table 3 above. The remaining six URNs (1004, 1005, ..., 1009) hit R007. However, since the same URN exists in both input and output data flows, it is determined to be an intermediate result and will be ignored during the analysis process.

[0096] Step S503: Construct the blood relationship tuples of database_a table_1 to database_c table_4, database_a table_2 to database_c table_4, and database_a table_3 to database_c table_4, and generate a snapshot of the collection time to obtain a blood relationship graph (such as Figure 4 shown).

[0097] The stock binary table of the above embodiment is shown in Table 4.

[0098] Table 4 Stock binary table

[0099] Source database Source data table Target database Target data table Collection time database_a table_1 database_c table_4 20231231 database_a table_5 database_c table_4 20231231 database_b table_3 database_c table_4 20231231

[0100] For example, the binary relationship between table_5 and table_4 in the existing binary table in Table 4 no longer appears in 20240101, and the binary relationship between table_2 and table_4 is newly added in the new binary table in Table 3. If the blood relationship is constructed with 20231231 as the time snapshot, the result will be Figure 2 If we use 20240101 as the time snapshot to build the blood relationship, we will get Figure 4 The full binary table after adding the binary relationship between table_2 and table_4 is shown in Table 5.

[0101] Table 5 Full 2-tuple table

[0102] Source database Source data table Target database Target data table Collection time database_a table_1 database_c table_4 20240101 database_a table_2 database_c table_4 20240101 database_b table_3 database_c table_4 20240101 database_a table_5 database_c table_4 20231231

[0103] The technical solution of this application constructs a data lineage system through the URN of data input and output streams, solves the relationship of data flow from the perspective of underlying data flow, and fundamentally solves the problem of unresolvable lineage relationships caused by different upper-level non-SQL applications. It does not rely on manual maintenance of lineage relationships, and the constructed lineage relationships are more accurate. In addition, lineage relationships can be traced back based on different time snapshots.

[0104] See Figure 6 , Figure 6 A data lineage construction system based on input and output URNs is provided, the system comprising:

[0105] The receiving unit 601 is used to receive a task to be processed, start executing MapTask for the task, and sequentially read the input data required by the task;

[0106] The analysis unit 602 is used to assemble a unified URN according to the preset input and output stream URN rules and the data information of the input data. When the task completes the MapTask, it parses the input stream URN and output stream URN of the MapTask to obtain the blood relationship tuple of the source subsystem, source database, source data table, target subsystem, target database, target data table, and collection time;

[0107] The construction unit 603 is used to write the blood relationship tuple into the blood relationship database to realize the construction of blood relationship.

[0108] For example, the above-mentioned analysis unit 602 is specifically used to match the data information of the input data with the preset input and output stream URN rules to obtain a rule ID number with the same information as the data information of the input data, and assemble the data information of the input data into a unified URN according to the arrangement order of the rule expression of the rule ID number.

[0109] For example, the above-mentioned analysis unit 602 is specifically used to obtain the file path, data protocol and database ID in the data information of the input data, and encapsulate the file path, data protocol and database ID into a unified URN according to the assembly order of the first URN rule corresponding to the rule ID number.

[0110] For example, the above-mentioned analysis unit 602 is specifically used to match the output stream URN corresponding to the input stream URN according to the input stream URN of MapTask, obtain the target database, target data table and target subsystem corresponding to the output stream URN, obtain the source subsystem, source database, source data table corresponding to the input stream URN, construct a system tuple from the source subsystem to the target subsystem, and construct a blood relationship tuple from the source database and source data table to the target database and target data table.

[0111] For example, the analysis unit 602 is specifically configured to extract a first rule ID number matched by the input stream URN, extract a source subsystem corresponding to the first rule ID number from the preset input and output stream URN rules, and extract a source database and a source data table from the input stream URN;

[0112] For example, the above-mentioned analysis unit 602 is specifically used to extract the second rule ID number matched by the output flow URN, extract the subsystem corresponding to the second rule ID number as the target subsystem in the preset input and output flow URN rules, and extract the target database and target data table from the output flow URN.

[0113] For example, the construction unit 603 is specifically configured to update the collection time if the blood relationship data already exists in the database; and insert the blood relationship into the blood relationship database if the blood relationship data does not exist.

[0114] For example, the above-mentioned construction unit 603 is specifically used to use the source database and source data table as the starting point of the data lineage graph, the target database and target data table as the end point of the data lineage graph, the association relationship as the edge of the data lineage graph, and the collection time as the attribute on the directed edge to construct a data lineage graph from the starting point to the end point.

[0115] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0116] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0117] The present application also provides a computer program product comprising a non-transitory computer-readable storage medium storing a computer program, wherein the computer program is operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may comprise an electronic device.

[0118] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices, and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is merely a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, the functional units in various embodiments of the present invention may be integrated into a single processing unit, each unit may be physically included separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional units.

[0122] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM), among other media that can store program code.

[0123] Although the present invention is disclosed above, it is not limited thereto. Any person skilled in the art may readily conceive of variations or substitutions, and may make various modifications and alterations without departing from the spirit and scope of the present invention. Combinations of the above-described functions and implementation steps, including software and hardware implementations, are all within the scope of protection of the present invention.

Claims

1. A data lineage construction method based on input and output URN, characterized in that: The method comprises the following steps: Receive the task to be processed, start executing MapTask for the task, and read the input data required by the task in sequence; According to the preset input and output stream URN rules and the data information of the input data, a unified URN is assembled. When the task completes the MapTask, the kinship tuple of the source subsystem, source database, source data table, target subsystem, target database, target data table, and collection time is obtained according to the input stream URN and output stream URN of the MapTask. The blood relationship tuple is written into the blood relationship database to realize the construction of blood relationship.

2. The data lineage construction method based on input and output URN according to claim 1 is characterized in that: The data information of the input data specifically includes: one or any combination of the IP address, port number, library name, table name, time, file path, and language type of the input stream data.

3. The data lineage construction method based on input and output URN according to claim 2 is characterized in that: The assembling of the preset input and output stream URN rules and the data information of the input data into a unified URN specifically includes: The data information of the input data is matched with the preset input and output stream URN rules to obtain a rule ID number having the same information as the data information of the input data, and the data information of the input data is assembled into a unified URN according to the arrangement order of the rule expression of the rule ID number.

4. The data lineage construction method based on input and output URN according to claim 3 is characterized in that: The step of assembling the data information of the input data into a unified URN according to the arrangement order of the rule expression of the rule ID number specifically includes: The file path, data protocol and database ID in the data information of the input data are obtained, and the file path, data protocol and database ID are encapsulated into a unified URN according to the assembly order of the first URN rule corresponding to the rule ID number.

5. The data lineage construction method based on input and output URN according to claim 1 is characterized in that: The step of assembling the data information of the input data into a unified URN according to the arrangement order of the rule expression of the rule ID number specifically includes: According to the input stream URN of MapTask, match the output stream URN corresponding to the input stream URN, obtain the target database, target data table and target subsystem corresponding to the output stream URN, obtain the source subsystem, source database and source data table corresponding to the input stream URN, construct a system tuple from the source subsystem to the target subsystem, and construct a blood relationship tuple from the source database and source data table to the target database and target data table.

6. The data lineage construction method based on input and output URN according to claim 5 is characterized in that: The method of obtaining the source subsystem, source database, and source data table corresponding to the input stream URN specifically includes: Extract the first rule ID number matched by the input stream URN, extract the source subsystem corresponding to the first rule ID number in the preset input and output stream URN rules, and extract the source database and source data table from the input stream URN; The target database, target data table and target subsystem corresponding to the output stream URN are obtained specifically including: Extract the second rule ID number matched by the output stream URN, extract the subsystem corresponding to the second rule ID number in the preset input and output stream URN rules as the target subsystem, and extract the target database and target data table from the output stream URN.

7. The data lineage construction method based on input and output URN according to claim 1 is characterized in that: The step of writing the blood relationship binary group into the blood relationship database to realize blood relationship construction specifically includes: If the blood relationship data already exists in the database, the collection time will be updated; if the blood relationship does not exist, the blood relationship will be inserted into the blood relationship database.

8. The data lineage construction method based on input and output URN according to claim 7 is characterized in that: The inserting of the blood relationship into the blood relationship database specifically includes: The source database and source data table are taken as the starting point of the data lineage graph, the target database and target data table are taken as the end point of the data lineage graph, the association relationship is taken as the edge of the data lineage graph, and the collection time is taken as the attribute on the directed edge to construct a data lineage graph from the starting point to the end point.

9. A data lineage construction system based on input and output URN, characterized in that: The system comprises: A receiving unit, used for receiving a task to be processed, starting to execute a MapTask for the task, and sequentially reading input data required for the task; The analysis unit is used to assemble a unified URN according to the preset input and output stream URN rules and the data information of the input data. When the task completes the execution of MapTask, the kinship tuple of the source subsystem, source database, source data table, target subsystem, target database, target data table and collection time is obtained according to the input stream URN and output stream URN of MapTask; The construction unit is used to write the blood relationship binary group into the blood relationship database to realize the construction of blood relationship.

10. A computer storage medium storing a computer program for electronic data exchange, wherein: The computer program enables a computer to execute part or all of the steps of the method according to any one of claims 1 to 8.

Images