Method and apparatus for acquiring lineage data, computer device, and readable storage medium

Abstract

Disclosed in embodiments of the present application are a method and apparatus for acquiring lineage data, a computer device, and a readable storage medium. The method comprises: obtaining lineage information of a real-time generation task and data information of a dependency relationship between the real-time generation task and a predecessor task thereof, the lineage information comprising at least two among identification information of the real-time generation task, a data input database table, a data output database table, an output relationship, and an input relationship, and the data information comprising at least two among identification information of the predecessor task, status, and a first run time; writing the lineage information and the data information into message middleware, and processing the lineage information and the data information in the message middleware, and obtaining a lineage map and an index map of the real-time generation task; and uploading the lineage map and the index map to a graph database and an index database, respectively. In the present application, comprehensive acquisition of lineage data can be implemented, so that impact analysis can be carried out on a data table in a lineage map.

Description

Bloodline data collection method, device, computer equipment and readable storage medium Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, apparatus, computer equipment and readable storage medium for collecting bloodline data. Background Art

[0002] Currently, the industry typically uses various Structured Query Language (SQL) tasks in Hive, Spark, or Presto to collect lineage data. However, the collected lineage data is heavily dependent on SQL tasks, resulting in incomplete lineage data collection. In addition, because lineage data is collected independently in units of SQL fragments of SQL tasks, important information between SQL fragments is ignored, making it impossible to perform impact analysis on data tables in the lineage map.

[0003] Summary of the Invention

[0004] In response to the shortcomings of related technologies, the embodiments of the present application provide a bloodline data collection method, device, computer equipment and readable storage medium, aiming to solve the technical problems in related technologies that bloodline data collection is incomplete and cannot perform impact analysis on data tables in bloodline maps.

[0005] To solve the above problems, in a first aspect, an embodiment of the present application provides a lineage data collection method, including: obtaining lineage information of a real-time generation task and data information of the dependency relationship between the real-time generation task and the predecessor task of the real-time generation task; wherein the lineage information includes at least two of the identification information of the real-time generation task, a data input library table, a data output library table, an output relationship and an input relationship, and the data information includes at least two of the identification information, status and first running time of the predecessor task, the data input library table is used to represent the data table input to the real-time generation task, the input relationship is used to represent the relationship between the data input library table and the real-time generation task, the data output library table is used to represent the data table output by the real-time generation task, the output relationship is used to represent the relationship between the data output library table and the real-time generation task, and the status is used to represent the running status of the predecessor task; writing the lineage information and data information into a preset message middleware, and processing the lineage information and data information in the message middleware to obtain a lineage map and an index map of the real-time generation task; uploading the lineage map and the index map to a preset graph database and an index database respectively.

[0006] In the second aspect, an embodiment of the present application also provides a lineage data collection device, including: a first acquisition unit, used to obtain the lineage information of the real-time generation task and the data information of the dependency relationship between the real-time generation task and the predecessor task of the real-time generation task; wherein the lineage information includes the identification information of the real-time generation task, the data input library table, the data output library table, the output relationship and at least two of the input relationship, the dependency relationship includes the identification information of the predecessor task, the status and at least two of the first running time, the data input library table is used to represent the data table input to the real-time generation task, the input relationship is used to represent the relationship between the data input library table and the real-time generation task, the data output library table is used to represent the data table output by the real-time generation task, the output relationship is used to represent the relationship between the data output library table and the real-time generation task, and the status is used to represent the running status of the predecessor task; a processing unit, used to write the lineage information and data information into a preset message middleware, and process the lineage information and data information in the message middleware to obtain the lineage map and index map of the real-time generation task; a first uploading unit, used to upload the lineage map and index map to a preset graph database and index database respectively.

[0007] In a third aspect, an embodiment of the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the bloodline data collection method as described in the first aspect above is implemented.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the bloodline data collection method described in the first aspect above.

[0009] An embodiment of the present application provides a method, apparatus, computer equipment and readable storage medium for collecting lineage data. In the method, the lineage information of a real-time generated task and data information of the dependency relationship between the real-time generated task and its predecessor task are obtained to achieve comprehensive collection of lineage data, and the lineage information and data information are consumed in a preset message middleware to obtain a lineage map and an index map of the real-time generated task, and the lineage map and the index map are uploaded to a graph database and an index database respectively, so that when performing impact analysis on a data table in the lineage map, the corresponding lineage map and index map can be directly obtained from the graph database and the index database. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIG1 is an architectural diagram of the bloodline data collection method provided in an embodiment of the present application.

[0011] FIG2 is a flow chart of the blood relationship data collection method provided in an embodiment of the present application.

[0012] FIG3 is a flow chart of a method for collecting bloodline data according to an embodiment of the present application.

[0013] FIG4 is another flow chart of the method for collecting bloodline data provided in an embodiment of the present application.

[0014] FIG5 is another flow chart of the blood relationship data collection method provided in an embodiment of the present application.

[0015] FIG6 is a block diagram of the blood relationship map in the blood relationship data collection method provided in an embodiment of the present application.

[0016] FIG7 is a block diagram of the blood relationship map in the blood relationship data collection method provided in an embodiment of the present application.

[0017] FIG8 is a block diagram of the index map in the bloodline data collection method provided in an embodiment of the present application.

[0018] FIG9 is a block diagram of the index map in the bloodline data collection method provided in an embodiment of the present application.

[0019] FIG10 is another flow chart of the method for collecting bloodline data provided in an embodiment of the present application.

[0020] FIG11 is a partial block diagram of the bloodline map in the bloodline data collection method provided in an embodiment of the present application.

[0021] FIG12 is another flow chart of the blood relationship data collection method provided in an embodiment of the present application.

[0022] FIG13 is a block diagram of an index map in the bloodline data collection method provided in an embodiment of the present application.

[0023] FIG14 is another flow chart of the blood relationship data collection method provided in an embodiment of the present application.

[0024] FIG15 is a block diagram of the fused atlas in the bloodline data collection method provided in an embodiment of the present application.

[0025] FIG16 is a schematic block diagram of a bloodline data collection device provided in an embodiment of the present application.

[0026] FIG17 is a schematic block diagram of a computer device provided in an embodiment of the present application.

[0027] FIG18 is a schematic block diagram of a storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] From generation to ETL processing, processing, integration, circulation, and eventual dissolution, any data naturally forms relationships. This relationship between data can be expressed in a similar way to interpersonal relationships in human society, known as data lineage. Data lineage is a concept in data governance that seeks to identify connections between related data during the process of data traceability. It is a logical concept.

[0029] Currently, the industry typically parses SQL fragments from various Structured Query Language (SQL) tasks in Hive, Spark, or Presto into an Abstract Syntax Tree (AST), then traverses the AST to obtain lineage data. Finally, the lineage data is processed to obtain lineage information to meet business needs.

[0030] However, the bloodline data collected by the above method is heavily dependent on SQL tasks, resulting in incomplete bloodline data collection. In addition, since bloodline data is collected independently in units of SQL fragments, important information between SQL fragments is ignored, making it impossible to perform impact analysis on data tables in the bloodline map.

[0031] To solve the above problems, the embodiments of the present application provide a bloodline data collection method, apparatus, computer equipment and readable storage medium. The bloodline data collection method, apparatus, computer equipment and readable storage medium will be introduced in detail below.

[0032] Please refer to Figure 1, which is a diagram illustrating the architecture of the bloodline data collection method provided in an embodiment of the present application. As shown in Figure 1, the bloodline data collection method of an embodiment of the present application is applied to a terminal device and executed by application software installed on the terminal device. The terminal device can be a terminal device with internet access, such as a smart bracelet, desktop computer, laptop computer, tablet computer, or mobile phone.

[0033] In the architecture diagram shown in Figure 1, it includes back-end service A1, front-end service A2, data storage layer A3, lineage data collection layer A4, lineage data import layer A5 and application layer lineage import A6. The backend service A1 mainly provides functions such as graph data search and impact analysis algorithm. At the same time, the backend service A1 also has a lineage reporting interface, which can provide lineage graph and index graph upload functions; the front-end service A2 is mainly responsible for initiating front-end queries of lineage data, providing human-computer interaction interface, graph data rendering and data visualization interface, etc.; the data storage layer A3 mainly includes graph databases (such as Nebula Graph database) and index databases (such as MYSQL database), and the index database is used as auxiliary query means such as secondary index; the lineage data collection layer A4 is mainly responsible for generating lineage data and task status related data of SQL tasks or non-SQL tasks in real time, and writing them to the corresponding message middleware (such as Kafka Topic). The lineage data of SQL tasks can be written from HiveHook in the form of HiveSQL to the BDP-TABLE-USAGE of Kafka Topic, and the dependency information can be written to Kafka from the Basic Data Platform (BDP), MYSQL and sch-job tables in the form of binlog events (i.e., logs). Topic's FXBSCH-SCH-JOB; the lineage data import layer A5 uses Flink real-time task sets to consume the lineage data and task status-related data generated by the lineage data collection layer A4, to convert the lineage data and task status-related data into lineage graphs and index graphs in real time, and upload the lineage graphs to the A3 graph database through the backend service A1, and write the index graphs into the A3 index database. The lineage data import layer A5 can import HIVE lineage graph data, field-level lineage graph data, task status-related data, Extract-Transform-Load (ETL) lineage graph data, and Activity On Edge Network (AOE network) that represents the edge of the impact analysis task. Each upper-layer application can import non-SQL task type application layer lineage data into the graph database and index database through the lineage data reporting interface in the backend service A1 at the application layer lineage import A6 according to its own business scenario.

[0034] It should be noted that the application scenario of the above embodiment is only an example. The services and scenarios described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. It is known to those skilled in the art that with the evolution of the system and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. The following are detailed descriptions. It should be noted that the order of description of the following embodiments does not limit the preferred order of the embodiments.

[0035] The following is a detailed description of the bloodline data collection method. Please refer to Figure 2, which is a flow chart of the bloodline data collection method provided in an embodiment of the present application.

[0036] As shown in FIG2 , the method includes the following steps S110 to S130 .

[0037] S110. Obtain the lineage information of the real-time generated task and the data information of the dependency relationship between the real-time generated task and the predecessor task of the real-time generated task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship and input relationship of the real-time generated task, and the data information includes at least two of the identification information, status and first running time of the predecessor task.

[0038] Lineage information includes the real-time generation task's identification information, data input tables, data output tables, output relationships, or input relationships. Identification information can be a task ID. By analyzing this lineage information, we can establish lineage relationships between data, thereby tracing the lineage of data and tracing the evolution of a piece of data throughout the data stream, enabling a better understanding and use of the data. The dependency relationship between a real-time generation task and its predecessor tasks means that the real-time generation task depends on the predecessor task. This means that the real-time generation task can only run after the predecessor task has completed. The status of the predecessor task can include running, successful, or failed. Real-time generation tasks can be SQL tasks or non-SQL tasks, which can be collected and acquired through the entire chain. Non-SQL tasks can be HIVE ETL tasks, which define the rules for extracting, transforming, and loading data.

[0039] In an exemplary embodiment, when the real-time generation task includes an SQL task, as shown in FIG3 , step S110 includes:

[0040] S111, obtaining the SQL statement of the real-time generation SQL task and the first log of the real-time scheduling SQL task;

[0041] S112, parsing the SQL statement to obtain the lineage information of the SQL task;

[0042] S113 . According to the first log, obtain data information of the dependency relationship between the SQL task and its predecessor task from a relational database for real-time scheduling of SQL tasks.

[0043] In this embodiment, the real-time generation task includes an SQL task. When collecting the lineage information of the SQL task and the dependency relationship with the predecessor task, the SQL statement of the real-time generation SQL task and the first log of the real-time scheduling SQL task are obtained, and in the computing engine (Hive or Spark, etc.), the SQL statement is parsed into AST syntax data through plug-ins, Hook functions or source code modification. Then, the AST syntax data is traversed to obtain the SQL task's identification information, data input library tables, data output library tables, output relations, and input relations and other lineage information, and then the parsed lineage information can be written into the preset message middleware in a specific format. The first log is the Binlog log of the relational database that monitors the scheduling task. After obtaining the first log, the relational database of the real-time scheduling SQL task can be reversely checked based on the first log to obtain the identification information, status and first running time of the predecessor task of the SQL task.

[0044] In other application embodiments, when the real-time generated tasks include non-SQL tasks, as shown in FIG4 , step S110 includes:

[0045] S1101, obtaining the second log of the real-time scheduling non-SQL task;

[0046] S1102 : According to the second log, obtain the lineage information of the non-SQL task and the data information of the dependency relationship between the non-SQL task and its predecessor task from the relational database for real-time scheduling of the non-SQL task.

[0047] In this embodiment, the real-time generated task is a non-SQL task. When collecting the lineage information of the non-SQL task and the dependency relationship with the predecessor task, the second log of the real-time scheduled non-SQL task can be directly obtained to realize the collection of the lineage information of the non-SQL task and the dependency relationship between the non-SQL task and the predecessor task of the non-SQL task. The second log is also the Binlog log of the relational database for monitoring and scheduling tasks. After obtaining the second log, the relational database for real-time scheduling of the non-SQL task is reversely checked according to the second log to obtain the identification information, data input library table, data output library table, output relationship, input relationship and other lineage information of the non-SQL task and the data information of the dependency relationship with the predecessor task.

[0048] S120, writing the lineage information and dependency relationship into the preset message middleware, and processing the lineage information and data information in the message middleware to obtain the lineage map and index map of the real-time generated task.

[0049] After obtaining the lineage information of the real-time generated task and the data information on the dependency relationship between the real-time generated task and its predecessor task (also known as dependency information), the lineage information and dependency information can be written to the pre-configured message middleware (such as Kafka Topic) in a specific format. By consuming the lineage information and dependency information in the message middleware, a lineage graph and index graph of the real-time generated task can be generated. A lineage graph refers to a lineage relationship graph similar to that of human society formed between data during the process of data generation, processing, flow and extinction. Both the lineage graph and the index graph are directed graphs composed of nodes and edges between nodes.

[0050] In an exemplary embodiment, as shown in FIG5 , step S120 includes:

[0051] S121. Construct multiple library table nodes of the bloodline map based on the data input library table and the data output library table;

[0052] S122: constructing a plurality of task nodes in a bloodline graph and a plurality of task nodes in an index graph according to the identification information of the real-time generated task and the identification information of the predecessor task;

[0053] S123. Construct a first connection relationship between the library table node and the task node in the bloodline graph based on the output relationship and the input relationship;

[0054] S124. Construct a second connection relationship between each task node in the index graph based on the data information;

[0055] S125, constructing a bloodline map based on the multiple library table nodes, the multiple task nodes in the bloodline map, and the first connection relationship;

[0056] S126. Construct an index graph according to the multiple task nodes and the second connection relationship of the index graph.

[0057] The nodes in the lineage map are library table nodes and task nodes. The library table nodes are used to represent database tables, and the task nodes are used to represent tasks. The connection between the library table nodes and the task nodes is used to represent the first connection relationship between the library table nodes and the task nodes. The connection is a directed connection. The database table includes a data input library table and a data output library table. The data input library table is a data table for inputting into the task. The data input library table and the task have an input relationship. The data output library table is a data table for outputting the task. The data output library table and the task have an output relationship. The first connection relationship includes an input relationship and an output relationship. The nodes in the index map are task nodes, which are composed of a task and its predecessor task. The connection between each task node in the index map is used to represent the second connection relationship between the task nodes. The connection is also a directed connection. The task not pointed to by the directed connection serves as the predecessor task of another task. The relationship between two tasks (for example, a task and its predecessor task) is characterized by the running time after the predecessor task is completed. The first running time can be the average time of multiple runs of the predecessor task.

[0058] In this embodiment, after the lineage information and dependency information are written into the message middleware, a Flink real-time task is established in the lineage data import layer, and logical operations are performed on the lineage information and dependency information in the message middleware, so that the identification information of the data input library table, the data output library table and the real-time generation task can be converted into the library table node and task node in the lineage graph, the identification information of the real-time generation task and the identification information of the predecessor task are converted into the task node in the index graph, the input relationship and output relationship between the data input library table, the data output library table and the real-time generation task are respectively converted into the first connection relationship between the library table node and the task node in the lineage graph, the dependency information is converted into the second connection relationship between each task node in the index graph, the first connection relationship and the second connection relationship are represented by directed lines, and finally the library table node and the task node in the lineage graph are linked through the first connection relationship, and the task nodes in the index graph are linked through the second connection relationship, so that a lineage graph and an index graph can be constructed.

[0059] Please refer to Figure 6. By collecting the blood relationship information of a task with ID 100, a blood relationship graph is constructed. It can be regarded as an input / output (IO) model. Task 100 includes two SQL statements. One SQL statement includes Table A, Table B, and Table C. Table A and Table B can be used as data input tables in this SQL statement, and Table C can be used as a data output table in this SQL statement. The other SQL statement includes Table A, Table C, and Table D. Table A and Table C can be used as data input tables in this SQL statement, and Table D can be used as a data output table in this SQL statement. The input-output relationship represents the directed connections between Table A, Table B, Table C, and Table D and Task 100, respectively. Figure 6 only shows the blood relationship graph constructed by one task. As blood relationship data is collected, a blood relationship graph in the graph database shown in Figure 7 can be formed based on Figure 6. As can be seen from Figure 7, Table D is generated by Table E and Table F through Task 101 and Task 102, respectively.

[0060] Please refer to Figure 8, which shows an index graph constructed through the dependency relationship between a task and its predecessor tasks. The task ID is 100, and the predecessor task IDs are 98 and 99 respectively. Task 98, Task 99, and Task 100 are three task nodes in the index graph. The last 14 time consumptions of Task 98 and Task 99 serve as directed links between Task 98, Task 99, and Task 100. Similarly, with the collection of lineage data, an index graph in the index database shown in Figure 9 can be formed based on Figure 8. As can be seen from Figure 9, Task 100 is the predecessor task of Task X, and the last 14 time consumptions of Task 100 can serve as the directed link between Task 100 and Task X. Task Y is the predecessor task of Task 98, and the last 14 time consumptions of Task Y can serve as the directed link between Task Y and Task 98.

[0061] In an exemplary embodiment, as shown in FIG10 , step S125 includes:

[0062] S1251. Constructing a third connection relationship between each library table node in the bloodline graph based on the identification information of the real-time generated task;

[0063] S1252. Construct a bloodline map based on multiple library table nodes, multiple task nodes in the bloodline map, the first connection relationship, and the third connection relationship.

[0064] The kinship map constructed in step S125 only considers the input and output between the database table and the task, and cannot show the input and output relationships between the database tables. To remedy this shortcoming, after determining the task between the two database tables, this embodiment converts the identification information of the real-time generated task into a third connection relationship between the two database tables and represents it with a wired connection. Finally, the library table node and the task node in the kinship map are linked through the first connection relationship, and the database tables are linked through the third connection relationship, thereby constructing a kinship map that can display the input and output relationships between the database tables.

[0065] Please refer to Figure 11, which is a detailed model used to represent the connection relationship between each table node in the bloodline map. In Figure 11, Table C is generated by Table A and Table B through Task 100, and Table D is generated by Table A and Table C through Task 100.

[0066] S130: Upload the bloodline map and index map to the preset map database and index database respectively.

[0067] After constructing the bloodline map and index map, the bloodline map and index map can be uploaded to the preset graph database and index data respectively through the bloodline reporting interface in the backend service. The bloodline map does not need to be uploaded through the bloodline reporting interface in the backend service. The bloodline map can be uploaded to the graph database directly through the channel between the bloodline data import layer and the graph database.

[0068] In an exemplary embodiment, as shown in FIG12 , after step S130 , the bloodline data collection method further includes:

[0069] S140: Obtaining an upstream task for generating a database table to be analyzed from a graph database; wherein the task node of the upstream task is connected to the database table node of the database table to be analyzed;

[0070] S150, obtaining the index map of the upstream task from the index database;

[0071] S160: Obtain a second running time for running the upstream task to generate the database table to be analyzed;

[0072] S170: Determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed based on the index map of the upstream task and the second running time.

[0073] In this embodiment, when determining the longest time-consuming path or the most resource-consuming path to generate the database table to be analyzed, the database table to be analyzed is used as the starting point, and the upstream task connected to the database table to be analyzed is obtained from the lineage map of the graph database, and the index map of the upstream task is obtained from the index map in the index database through the upstream task, and the second running time of running the upstream task to generate the database table to be analyzed is obtained, so that the longest running time for generating the upstream task and the second running time for running the upstream task to generate the database table to be analyzed can be obtained from the index map through the upstream task, and then the longest time-consuming path or the most resource-consuming path of the database table to be analyzed can be determined.

[0074] When obtaining the upstream tasks that generate the database table to be analyzed, if the graph database is Nebula Graph, you can use the GO FROM statement or GET SUBGRAPH statement, starting with the database table to be analyzed, to obtain the upstream tasks that generate the database table to be analyzed from Nebula Graph. For example, if the database table to be analyzed is Table D in Figure 7, you can determine from Figure 7 that the upstream tasks are Task 100, Task 101, and Task 102.

[0075] When retrieving the index graph of the upstream task from the Nebula Graph database, if the index database is a Nebula Graph database, you can also use the GO FROM statement or GET SUBGRAPH statement, starting with the upstream task, to retrieve the index graph from the Nebula Graph database. If the upstream tasks are tasks 100, 101, and 102 in Figure 7, the index graph of the upstream tasks can be seen in Figure 13. In Figure 13, the downstream task of task 100 is task X, and the upstream tasks of task 100 are tasks 98 and 99. The upstream task of task 98 is task Y, and the upstream task of task 101 is task 98.

[0076] In addition, regarding the acquisition of the second running time, it is obtained from the log of calling the upstream task to generate the database table to be analyzed. By parsing the log, the second running time of the upstream task that most recently ran to generate the database table to be analyzed is obtained. The second running time can be the average time of the upstream task that has recently run multiple times to generate the database table to be analyzed.

[0077] In an exemplary embodiment of the application, as shown in FIG14 , step S170 includes:

[0078] S171, fusing the to-be-analyzed database table, the second running time, and the index graph of the upstream task to obtain a fused graph;

[0079] S172. Using a critical path algorithm, determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed from the fused graph.

[0080] In this embodiment, in the process of determining the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed through the index graph of the upstream task and the second running time, all the nodes of the downstream task of the upstream task that generates the database table to be analyzed can be replaced with the nodes of the database table to be analyzed, and the connection relationship between the upstream task and the downstream task that generates the database table to be analyzed can be replaced with the second running time, so that the database table to be analyzed, the second running time and the index graph of the upstream task can be fused to obtain the fused graph as shown in Figure 15, and finally the critical path algorithm is run to process the fused graph, so that the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed can be determined.

[0081] In the lineage data collection method provided in the embodiment of the present application, the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task are obtained to realize comprehensive collection of lineage data; the lineage information and dependency relationship are written into the preset message middleware, and the lineage information and dependency relationship in the message middleware are processed to obtain the lineage map and index map of the real-time generated task; the lineage map and index map are uploaded to the preset graph database and index database respectively so that when performing impact analysis on the data table in the lineage map, the corresponding lineage map and index map can be directly obtained from the graph database and index database, so that the longest time-consuming link or the most resource-consuming link of the entire task network upstream of the task where the data table is located can be tracked, thereby realizing time optimization and task resource optimization of the data table.

[0082] The method embodiment of the present application is described in detail above in conjunction with Figures 1 to 15 , and the device embodiment of the present application is described in detail below in conjunction with Figures 16 and 17 . It should be understood that the description of the method embodiment corresponds to the description of the device embodiment, and therefore, for portions not described in detail, reference can be made to the above method embodiment.

[0083] The embodiment of the present application also provides a bloodline data collection device 100, which is used to execute any embodiment of the aforementioned bloodline data collection method.

[0084] Specifically, please refer to Figure 16, which is a schematic block diagram of the bloodline data collection device 100 provided in an embodiment of the present application.

[0085] As shown in FIG16 , the bloodline data collection device 100 includes: a first acquisition unit 110 , a processing unit 120 and a first uploading unit 130 .

[0086] The first acquisition unit 110 is used to obtain the lineage information of the real-time generation task and the data information of the dependency relationship between the real-time generation task and the predecessor task of the real-time generation task; wherein the lineage information includes the identification information of the real-time generation task, the data input library table, the data output library table, the output relationship and at least two of the input relationship; the data information includes at least two of the identification information, status and first running time of the predecessor task; the data input library table is used to represent the data table for input to the real-time generation task; the input relationship is used to represent the relationship between the data input library table and the real-time generation task; the data output library table is used to represent the data table output by the real-time generation task; the output relationship is used to represent the relationship between the data output library table and the real-time generation task; and the status is used to represent the running status of the predecessor task.

[0087] The processing unit 120 is used to write the lineage information and data information into the preset message middleware, and process the lineage information and data information in the message middleware to obtain the lineage map and index map of the real-time generated task.

[0088] The first uploading unit 130 is used to upload the bloodline graph and the index graph to a preset graph database and an index database respectively.

[0089] In an exemplary embodiment, the real-time generation task includes a structured query language SQL task, and the first uploading unit 130 includes a second acquiring unit, a parsing unit, and a third acquiring unit.

[0090] The second acquisition unit is used to acquire the SQL statement of the real-time generated SQL task and the first log of the real-time scheduled SQL task.

[0091] The parsing unit is used to parse the SQL statement and obtain the lineage information of the SQL task.

[0092] The third acquisition unit is configured to acquire, according to the first log, data information of dependency relationships between SQL tasks and their predecessor tasks from a relational database for real-time scheduling of SQL tasks.

[0093] In an exemplary embodiment, the real-time generation task includes a non-SQL task, and the first uploading unit 130 includes a fourth obtaining unit and a fifth obtaining unit.

[0094] The fourth acquiring unit is used to acquire a second log of the real-time scheduling non-SQL task.

[0095] The fifth acquiring unit is configured to acquire, based on the second log, lineage information of the non-SQL tasks and data information of dependency relationships between the non-SQL tasks and their predecessor tasks from a relational database for real-time scheduling of non-SQL tasks.

[0096] In an exemplary embodiment, the processing unit 120 includes a first building unit, a second building unit, a third building unit, a fourth building unit, a fifth building unit, and a sixth building unit.

[0097] The first construction unit is used to construct multiple library table nodes of the bloodline map according to the data input library table and the data output library table.

[0098] The second construction unit is used to construct multiple task nodes in the bloodline map and multiple task nodes in the index map according to the identification information of the real-time generated task and the identification information of the predecessor task.

[0099] The third construction unit is used to construct a first connection relationship between the library table node and the task node in the bloodline graph according to the output relationship and the input relationship.

[0100] The fourth construction unit is used to construct a second connection relationship between each task node in the index graph according to the data information.

[0101] The fifth construction unit is used to construct a bloodline map according to multiple library table nodes, multiple task nodes in the bloodline map, and the first connection relationship.

[0102] The sixth construction unit is used to construct an index map according to the multiple task nodes and the second connection relationship of the index map.

[0103] In an exemplary embodiment, the fifth building unit includes a sixth building unit and a seventh building unit.

[0104] The sixth construction unit is used to construct a third connection relationship between the library table nodes in the bloodline graph according to the identification information of the real-time generation task.

[0105] The seventh construction unit is used to construct a bloodline map based on multiple library table nodes, multiple task nodes in the bloodline map, the first connection key and the third connection relationship.

[0106] In an exemplary embodiment, the bloodline data collection device 100 further includes a sixth acquisition unit, a seventh acquisition unit and a first determination unit.

[0107] The sixth acquisition unit is used to acquire an upstream task for generating a database table to be analyzed from the graph database; wherein the task node of the upstream task is connected to the database table node of the database table to be analyzed.

[0108] The seventh acquisition unit is used to acquire the index map of the upstream task from the index database.

[0109] The eighth acquiring unit is configured to acquire a second running time for running the upstream task to generate the database table to be analyzed.

[0110] The first determining unit is used to determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed according to the index map of the upstream task and the second running time.

[0111] In an exemplary embodiment, the first determining unit includes: a fusion unit and a second determining unit.

[0112] The fusion unit is used to fuse the database table to be analyzed, the second running time and the index map of the upstream task to obtain a fused map.

[0113] The second determining unit is used to use a critical path algorithm to determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed from the fused graph.

[0114] The blood relationship data collection device shown in FIG16 can also exist in the form of other unit components, which is not limited here, as long as the corresponding functions can be achieved.

[0115] In addition, what is shown in FIG16 is only one embodiment of the bloodline data collection device, not all embodiments. Based on the embodiment of the bloodline data collection device in this application, all other embodiments obtained by those skilled in the art without making any creative work are within the scope of protection of this application.

[0116] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned bloodline data acquisition device 100 and each unit can refer to the corresponding description in the aforementioned method embodiment. For the convenience and brevity of the description, it will not be repeated here.

[0117] The above-mentioned bloodline data collection device can be implemented in the form of a computer program, which can be run on a computer device as shown in Figure 17.

[0118] Please refer to Figure 17, which is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 can be a terminal, which can be a computer device with communication functions such as a smartphone, tablet computer, laptop computer, desktop computer, personal digital assistant, and wearable device.

[0119] 17 , the computer device 500 includes a processor 502 , a memory, and a network interface 505 connected via a system bus 501 . The memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0120] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, which, when executed, can enable the processor 502 to execute the bloodline data collection method in any of the above embodiments.

[0121] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500.

[0122] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a bloodline data collection method.

[0123] The network interface 505 is used to communicate with other devices over a network. Those skilled in the art will appreciate that the structure shown in FIG17 is merely a block diagram of a portion of the structure related to the present invention, and does not limit the computer device 500 to which the present invention is applied. A specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0124] The processor 502 is configured to run a computer program 5032 stored in the memory to implement the bloodline data collection method in any of the above embodiments.

[0125] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0126] In addition, FIG17 shows only one embodiment of a computer device, not all embodiments. All other embodiments obtained by those skilled in the art based on the computer device embodiment in this application without any creative work shall fall within the scope of protection of this application.

[0127] Those skilled in the art will appreciate that all or part of the steps in the methods of the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the steps of the bloodline data collection method of any of the above-described embodiments.

[0128] Therefore, referring to FIG. 18 , the present application further provides a storage medium 1800 . The storage medium 1800 may be a computer-readable storage medium. The storage medium stores a computer program 1810 , wherein the computer program includes program instructions. When executed by a processor, the program instructions cause the processor to execute the bloodline data collection method described in any of the above embodiments.

[0129] The storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0130] In addition, FIG18 shows only one embodiment of the storage medium, not all embodiments. All other embodiments obtained by those skilled in the art based on the storage medium embodiment in this application without any creative work are within the scope of protection of this application.

[0131] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and other division methods may be used in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not implemented.

[0133] The steps in the method of the embodiment of the present application can be adjusted in order, combined, and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0135] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for collecting blood relationship data, characterized in that: include: Obtaining the lineage information of the real-time generation task and the data information of the dependency relationship between the real-time generation task and the predecessor task of the real-time generation task; wherein the lineage information includes the identification information of the real-time generation task, the data input library table, the data output library table, the output relationship and at least two of the input relationship, the data information includes at least two of the identification information, status and first running time of the predecessor task, the data input library table is used to represent the data table input to the real-time generation task, the input relationship is used to represent the relationship between the data input library table and the real-time generation task, the data output library table is used to represent the data table output by the real-time generation task, the output relationship is used to represent the relationship between the data output library table and the real-time generation task, and the status is used to represent the running status of the predecessor task; Writing the blood relationship information and the data information into a preset message middleware, and processing the blood relationship information and the data information in the message middleware to obtain a blood relationship map and an index map of the real-time generated task; The blood relationship map and the index map are uploaded to a preset graph database and an index database respectively.

2. The method for collecting blood relationship data according to claim 1, characterized in that: The real-time generation task includes a structured query language SQL task; The step of acquiring the lineage information of the real-time generated task and the data information of the dependency relationship between the real-time generated task and the predecessor task of the real-time generated task includes: Acquire a SQL statement for generating the SQL task in real time and a first log for scheduling the SQL task in real time; Parsing the SQL statement to obtain the lineage information of the SQL task; According to the first log, data information of a dependency relationship between the SQL task and a predecessor task of the SQL task is obtained from a relational database for real-time scheduling of the SQL task.

3. The method for collecting blood relationship data according to claim 1, characterized in that: The real-time generation tasks include non-SQL tasks; The step of acquiring the lineage information of the real-time generated task and the data information of the dependency relationship between the real-time generated task and the predecessor task of the real-time generated task includes: Obtain a second log for real-time scheduling of the non-SQL task; According to the second log, the lineage information of the non-SQL task and the data information of the dependency relationship between the non-SQL task and the predecessor task of the non-SQL task are obtained from the relational database for real-time scheduling of the non-SQL task.

4. The method for collecting blood relationship data according to any one of claims 1 to 3, characterized in that: The processing of the lineage information and data information in the message middleware to obtain the lineage graph and index graph of the real-time generated task includes: Constructing a plurality of library table nodes of the blood relationship map according to the data input library table and the data output library table; According to the identification information of the real-time generated task and the identification information of the predecessor task, a plurality of task nodes in the bloodline graph and a plurality of task nodes in the index graph are respectively constructed; According to the output relationship and the input relationship, a first connection relationship is constructed between the library table node and the task node in the blood relationship map; Constructing a second connection relationship between task nodes in the index graph according to the data information; Constructing the bloodline map according to the multiple library table nodes, the multiple task nodes in the bloodline map, and the first connection relationship; The index map is constructed according to the multiple task nodes of the index map and the second connection relationship.

5. The method for collecting blood relationship data according to claim 4, characterized in that: The step of constructing the bloodline map according to the plurality of library table nodes, the plurality of task nodes in the bloodline map, and the first connection relationship includes: According to the identification information of the real-time generation task, a third connection relationship between the library table nodes in the blood relationship map is constructed; The bloodline map is constructed according to the multiple library table nodes, the multiple task nodes in the bloodline map, the first connection relationship and the third connection relationship.

6. The method for collecting blood relationship data according to any one of claims 1 to 5, characterized in that: Also includes: Acquire an upstream task for generating a database table to be analyzed from the graph database; wherein a task node of the upstream task is connected to a database table node of the database table to be analyzed; Acquire the index graph of the upstream task from the index database; Acquire a second running time for running the upstream task to generate the database table to be analyzed; According to the index map of the upstream task and the second running time, the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed is determined.

7. The method for collecting blood relationship data according to claim 6, characterized in that: The determining, according to the index map of the upstream task and the second running time, the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed includes: Fusing the to-be-analyzed database table, the second running time, and the index graph of the upstream task to obtain a fused graph; A critical path algorithm is used to determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed from the fused graph.

8. A blood relationship data collection device, characterized in that: include: A first acquisition unit is used to acquire the lineage information of the real-time generation task and the data information of the dependency relationship between the real-time generation task and the predecessor task of the real-time generation task; wherein the lineage information includes the identification information of the real-time generation task, the data input library table, the data output library table, the output relationship and at least two of the input relationship; the data information includes at least two of the identification information, status and first running time of the predecessor task; the data input library table is used to represent the data table input to the real-time generation task; the input relationship is used to represent the relationship between the data input library table and the real-time generation task; the data output library table is used to represent the data table output by the real-time generation task; the output relationship is used to represent the relationship between the data output library table and the real-time generation task; and the status is used to represent the running status of the predecessor task; A processing unit, used for writing the blood relationship information and the data information into a preset message middleware, and processing the blood relationship information and the data information in the message middleware to obtain a blood relationship map and an index map of the real-time generated task; The first uploading unit is used to upload the blood relationship map and the index map to a preset map database and index database respectively.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the bloodline data collection method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to perform the bloodline data collection method according to any one of claims 1 to 7.

Images