Blood relationship data acquisition method and device, computer equipment and readable storage medium

Abstract

The embodiment of the invention discloses a blood relationship data acquisition method and device, computer equipment and a readable storage medium. The method comprises the following steps: acquiring blood relationship information of a real-time generation task and a dependency relationship between the real-time generation task and a preposed task; the margin information comprises at least two of identification information, a data input library table, a data output library table, an output relationship and an input relationship of the real-time generation task, and the dependency relationship comprises at least two of identification information, a state and first running time of the front task; writing the blood relationship information and the dependency relationship into message middleware, and processing the blood relationship information and the dependency relationship in the message middleware to obtain a blood relationship map and an index map of the real-time generation task; and respectively uploading the blood relationship map and the index map to a map database and an index database. According to the invention, comprehensive collection of blood relationship data can be realized, and aging optimization and task resource optimization of the data table can be realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a bloodline data collection method, device, computer equipment and readable storage medium. Background Art

[0002] At present, the industry usually parses the SQL fragments of various SQL (Structured Query Language) tasks in Hive, Spark, and Presto into AST (Abstract Syntax Tree) syntax trees, traverses the syntax trees to obtain lineage data, and then processes the lineage data to obtain lineage information to meet business needs.

[0003] However, the bloodline data collected by the above method is heavily dependent on SQL tasks, resulting in incomplete bloodline data collection. At the same time, since the bloodline data is collected independently in units of SQL fragments, the important information between SQL fragments is ignored, resulting in the inability to perform impact analysis on the data tables in the bloodline map and resource optimization of the tasks corresponding to the data tables. Summary of the invention

[0004] In view of the shortcomings of the prior art, the embodiments of the present invention provide a bloodline data collection method, apparatus, computer equipment and readable storage medium, aiming to solve the technical problems in the prior art that bloodline data collection is incomplete and cannot realize impact analysis of data tables in bloodline maps and resource optimization of tasks corresponding to the data tables.

[0005] To solve the above problems, in a first aspect, an embodiment of the present invention provides a method for collecting blood relationship data, which includes:

[0006] Obtaining lineage information of a real-time generated task and a dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship, and input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, state, and first running time of the predecessor task;

[0007] Writing the lineage information and the dependency relationship into a preset message middleware, and processing the lineage information and the dependency relationship in the message middleware to obtain a lineage graph and an index graph of the real-time generated task;

[0008] The blood relationship map and the index map are uploaded to a preset map database and an index database respectively.

[0009] In a second aspect, an embodiment of the present invention further provides a blood relationship data collection device, which includes:

[0010] A first acquisition unit is used to acquire the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship and input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, state and first running time of the predecessor task;

[0011] A processing unit, used for writing the blood relationship information and the dependency relationship into a preset message middleware, and processing the blood relationship information and the dependency relationship in the message middleware to obtain a blood relationship map and an index map of the real-time generated task;

[0012] The first uploading unit is used to upload the blood relationship map and the index map to a preset map database and an index database respectively.

[0013] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the bloodline data collection method as described in the first aspect above is implemented.

[0014] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the bloodline data collection method described in the first aspect above.

[0015] An embodiment of the present invention provides a lineage data collection method, apparatus, computer equipment and readable storage medium. The method realizes comprehensive collection of lineage data by acquiring the lineage information of real-time generated tasks and the dependency relationship between them and predecessor tasks, and consumes the acquired lineage information and dependency relationship in a preset message middleware to obtain a lineage map and an index map of the real-time generated tasks, and uploads them to a graph database and an index database respectively. When performing an impact analysis on a data table in the lineage map, the corresponding lineage map and index map can be directly obtained from the graph database and the index database, so that the longest time-consuming link or the most resource-consuming link of the entire task network upstream of the task where the data table is located can be tracked, thereby realizing timeliness optimization and task resource optimization of the data table. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0017] Figure 1 An architectural diagram of a blood relationship data collection method provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a process for collecting blood relationship data provided by an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of a process of collecting blood relationship data provided by an embodiment of the present invention;

[0020] Figure 4 Another schematic diagram of the process of collecting blood relationship data provided by an embodiment of the present invention;

[0021] Figure 5 Another schematic diagram of the process of collecting blood relationship data provided by an embodiment of the present invention;

[0022] Figure 6 A block diagram of a blood relationship map in a blood relationship data collection method provided in an embodiment of the present invention;

[0023] Figure 7 A block diagram of a blood relationship map in a blood relationship data collection method provided in an embodiment of the present invention;

[0024] Figure 8 A block diagram of an index map in a blood relationship data collection method provided in an embodiment of the present invention;

[0025] Fig. 9 A block diagram of an index map in a blood relationship data collection method provided in an embodiment of the present invention;

[0026] Fig.10 Another schematic diagram of the process of collecting blood relationship data provided by an embodiment of the present invention;

[0027] Fig.11 A partial block diagram of a blood relationship map in a blood relationship data collection method provided by an embodiment of the present invention;

[0028] Fig.12 Another schematic diagram of the process of collecting blood relationship data provided by an embodiment of the present invention;

[0029] Fig.13 A block diagram of an index graph in a blood relationship data collection method provided in an embodiment of the present invention;

[0030] Fig.14 Another schematic diagram of the process of collecting blood relationship data provided by an embodiment of the present invention;

[0031] Fig.15 A block diagram of a fused atlas in a blood relationship data collection method provided in an embodiment of the present invention;

[0032] Fig.16 Schematic block diagram of a blood relationship data collection device provided by an embodiment of the present invention

[0033] Fig.17 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0036] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0037] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0038] See also Figure 1 , Figure 1 This is a diagram of the architecture of the blood relationship data collection method provided by the embodiment of the present invention. Figure 1 As shown, the blood relationship data collection method of the embodiment of the present invention is applied to a terminal device, and the method is executed by an application software installed in the terminal device to generate a dense optical flow of an image. The terminal device can be a terminal device with an Internet access function, such as a smart bracelet, a desktop computer, a laptop computer, a tablet computer, or a mobile phone.

[0039] exist Figure 1 In the architecture diagram shown, it includes backend service A1, frontend service A2, data storage layer A3, lineage data collection layer A4, lineage data import layer A5 and application layer lineage import A6. Among them, the backend service A1 mainly provides functions such as graph data search and impact analysis algorithm. At the same time, the backend service A1 also has a lineage reporting interface, which can provide lineage map and index map upload functions; the front-end service A2 is mainly responsible for initiating front-end query of lineage data, providing human-computer interaction interface, graph data rendering and data visualization interface; the data storage layer A3 mainly includes graph database (such as Nebula Graph library) and index database (such as MYSQL library), and the index database is used as auxiliary query means such as secondary index; the lineage data collection layer A4 is mainly responsible for real-time generation of lineage data and task status related data of SQL tasks or non-SQL tasks, and writes them into the corresponding message middleware (such as Kafka Topic). Among them, the lineage data of SQL tasks can be written from HiveHook to BDP-TABLE-USAGE of Kafka Topic in the form of HiveSQL, and the dependency information can be written to Kafka in the form of binlog events, that is, logs, from the big data basic platform BDP, MYSQL and sch-job tables. Topic's FXBSCH-SCH-JOB; the lineage data import layer A5 uses Flink real-time tasks to consume the lineage data and task status data generated in the lineage data collection layer A4, and converts them into lineage graphs and index graphs in real time, and uploads the lineage graph to the Nebula graph database in A3 through the backend service A1, and writes the index graph into the MYSQL database in A3. The lineage data import layer A5 can import HIVE lineage graph data, field-level lineage graph data, task status, ETL lineage graph data, and impact analysis task AOE network data; each upper-layer application can import the application layer lineage data of non-SQL task types into the graph database and index database through the lineage data reporting interface in the backend service A1 at the application layer lineage import A6 according to its own business scenarios.

[0040] It should be noted that the application scenario of the above embodiment is only an example. The services and scenarios described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. It is known to those skilled in the art that with the evolution of the system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems. The following are detailed descriptions. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0041] The following is a detailed description of the blood relationship data collection method. Figure 2 , Figure 2 A flow chart of a method for collecting bloodline data provided in an embodiment of the present invention.

[0042] like Figure 2 As shown, the method includes the following steps S110 to S130.

[0043] S110. Obtain the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship and input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, status and first running time of the predecessor task.

[0044] Specifically, the lineage information consists of the identification information of the real-time generated task, the data input library table, the data output library table, the output relationship and the input relationship. The identification information can be the task ID. By analyzing the lineage information, the lineage relationship between the data can be established, and then the lineage of the data can be tracked, and the evolution process of a certain data in the entire data stream can be traced, so as to better understand and use the data. The dependency relationship between the real-time generated task and its predecessor task refers to the relationship that the operation of the task depends on the predecessor task, that is, the task can only be run after the predecessor task is completed. The status of the predecessor task includes running, success, failure, etc. The real-time generated task can be a SQL task or a non-SQL task, which can be collected and acquired through the whole link. The non-SQL task can be HIVE's ETL (Extract-Transform-Load) task. The ETL task is used to define the data extraction, transformation and loading rules.

[0045] In other inventive embodiments, Figure 3 As shown, step S110 includes steps S111, S112 and S113.

[0046] S111, obtaining a SQL statement for generating the SQL task in real time and a first log for scheduling the SQL task in real time;

[0047] S112, parsing the SQL statement of the SQL task to obtain the lineage information of the SQL task;

[0048] S113. Obtaining, according to the first log, a dependency relationship between the SQL task and its predecessor task from a relational database for real-time scheduling of the SQL task.

[0049] In this embodiment, the real-time generated tasks include SQL tasks. When collecting the lineage information and the dependency relationship between the SQL type tasks and the predecessor tasks, the SQL statements of the real-time generated SQL tasks and the first log of the real-time scheduled SQL tasks are obtained, and in the computing engine (Hive, Spark, etc.), the SQL statements are parsed into AST syntax data through plug-ins, Hook functions or source code modification, and then the AST syntax data is traversed to obtain the SQL task's identification information, data input library tables, data output library tables, output relations, input relations and other lineage information, and then the parsed lineage information can be written into the preset message middleware in a specific format. The first log is the Binlog log of the relational database that monitors the scheduling tasks. After obtaining the first log, the relational database of the real-time scheduled SQL tasks can be reversed according to the first log to obtain the identification information, status and first running time of the predecessor tasks of the SQL tasks.

[0050] In other inventive embodiments, Figure 4 As shown, step S110 includes steps S1101 and S1102.

[0051] S1101, obtaining a second log of real-time scheduling of the non-SQL task;

[0052] S1102: Acquire, according to the second log, the lineage information of the non-SQL task and the dependency relationship between the non-SQL task and its predecessor task from a relational database for real-time scheduling of the non-SQL task.

[0053] In this embodiment, the real-time generated task is a non-SQL task. When collecting the lineage information of the non-SQL type task and the dependency relationship with the predecessor task, the second log of the real-time scheduled non-SQL task can be directly obtained to realize the collection of the lineage information of the non-SQL task and the dependency relationship with the predecessor task. The second log is also the Binlog log of the relational database for monitoring and scheduling tasks. After obtaining the second log, the relational database of the real-time scheduled non-SQL task is reversed according to the second log to obtain the identification information, data input library table, data output library table, output relationship, input relationship and other lineage information of the non-SQL task and the data information of the dependency relationship with the predecessor task.

[0054] S120, writing the lineage information and the dependency relationship into a preset message middleware, and processing the lineage information and dependency relationship in the message middleware to obtain a lineage graph and an index graph of the real-time generated task.

[0055] Specifically, after obtaining the lineage information of the real-time generated task and the dependency information between the task and the predecessor task, the lineage information and dependency information can be written into the preset message middleware (such as Kafka Topic) in a specific format. By consuming the lineage information and dependency information in the message middleware, the lineage graph and lineage graph of the task can be generated. Among them, the lineage graph refers to a kind of lineage relationship graph similar to the human society formed between data in the process of data generation, processing, flow and extinction. The lineage graph and index graph are both directed graphs, which are composed of nodes and edges between nodes.

[0056] In other inventive embodiments, Figure 5 As shown, step S120 includes steps S121, S122, S123, S124, S125 and S126.

[0057] S121, constructing a plurality of library table nodes of the blood relationship map according to the data input library table and the data output library table;

[0058] S122, constructing the blood relationship map and a plurality of task nodes in the index map respectively according to the identification information of the real-time generated task and the identification information of the predecessor task;

[0059] S123, constructing a first connection relationship between a library table node and a task node in the bloodline graph according to the output relationship and the input relationship;

[0060] S124, constructing a second connection relationship between the task nodes in the index graph according to the dependency relationship;

[0061] S125, constructing the bloodline map according to the multiple library table nodes, multiple task nodes and the first connection relationship of the bloodline map;

[0062] S126. Construct the index graph according to the multiple task nodes and the second connection relationship of the index graph.

[0063] Specifically, the nodes in the bloodline map are library table nodes and task nodes. The library table nodes are used to represent database tables, and the task nodes are used to represent tasks. The connection between the library table nodes and the task nodes is used to represent the first connection relationship between the library table nodes and the task nodes. The connection is a directed connection. The database table includes a data input library table and a data output library table. The data input library table is a data table for inputting into the task. The input database table and the task have an input relationship. The data output library table is a data table for outputting the task. The output library table and the task have an output relationship. The first connection relationship includes an input relationship and an output relationship. The nodes in the index map are task nodes, which are composed of tasks and the predecessor tasks of the tasks. The connection between the task nodes in the index map is used to represent the second connection relationship between the task nodes. The connection is also a directed connection. The task not pointed to by the directed connection serves as the predecessor task of another task. The relationship between the two tasks is represented by the running time after the predecessor task is completed. The first running time can be the average time of multiple runs of the predecessor task.

[0064] In this embodiment, after the lineage information and dependency relationships are written into the message middleware, a Flink real-time task is established in the lineage data import layer, and logical operations are performed on the lineage information and dependency relationships of the tasks in the middleware. In this way, the data input library table, data output library table and task identification information in the lineage information can be converted into library table nodes and task nodes in the lineage graph, the task identification information in the lineage information and the predecessor task are converted into task nodes in the index graph, the input relationship and output relationship between the data input library table, data output library table and task in the lineage information are converted into the first connection relationship between the library table node and the task node in the lineage graph, the dependency relationship is converted into the second connection relationship between the task nodes in the index graph, the first connection relationship and the second connection relationship are represented by directed lines, and finally the library table node and the task node in the lineage graph are linked through the first connection relationship, and the task nodes in the index graph are linked through the second connection relationship, so that the lineage graph and the index graph can be constructed.

[0065] See also Figure 6 , Figure 6 In order to construct a blood relationship map by collecting blood relationship information of a task with taskId 100, it can be regarded as an IO model. The task consists of two SQL statements, one of which includes table A, table B and table C. Table A and table B can be used as data input library tables in the SQL statement, and table C can be used as data output library table in the SQL statement; the other SQL statement includes table A, table C and table D. Table A and table C are used as data input library tables in the SQL statement, and table D is used as data output library table in the SQL statement; iorelation represents the directed connection between table A, table B, table C and table D and the task respectively. Among them, Figure 6 The bloodline graph obtained by just one task is as follows: With the collection of bloodline data, the bloodline graph in the graph database can be Figure 6 Based on the formation of Figure 7 The blood relationship diagram shown is from Figure 7 It can be seen that table D is generated by table E and table F through task 101 and task 102 respectively.

[0066] For details, please refer to Figure 8 , Figure 8 The index graph is constructed through the dependency relationship between a task and its predecessor task. The task ID of the task is task100, and the IDs of the predecessor tasks are task98 and task99. Task98, task99 and task100 are the three task nodes of the index graph. The last 14 time consumptions of task98 and task99 are the directed links between task98, task99 and task100. Similarly, with the collection of lineage data, the index graph in the index database can be Figure 8 Based on the formation of Fig. 9 The index map shown is from Fig. 9 It can be seen that task100 is the predecessor task of taskX, and the recent 14 time consumptions of task100 can be used as a directed line between task100 and taskX. TaskY is the predecessor task of task98, and the recent 14 time consumptions of taskY can be used as a directed line between taskY and task98.

[0067] In other inventive embodiments, Fig.10 As shown, step S125 includes steps S1251 and S1252.

[0068] S1251, constructing a third connection relationship between each library table node in the blood relationship map according to the identification information of the real-time generation task;

[0069] S1252. Construct the bloodline map according to the multiple library table nodes, multiple task nodes, the first connection key and the third connection relationship of the bloodline map.

[0070] Specifically, the blood relationship map constructed in step S125 only considers the input and output between the database table and the task, and it is impossible to see the input relationship and output relationship between the database tables. To make up for this defect, after determining the task between the two database tables, this embodiment converts the identification information of the task into a third connection relationship between the two database tables, and represents it with a wired connection. Finally, the library table node and the task node in the blood relationship map are linked through the first connection relationship, and the database tables are linked through the third connection relationship, so that a blood relationship map that can display the input relationship and output relationship between the database tables can be constructed.

[0071] See also Fig.11 , Fig.11 It is a detailed model, which is used to represent the connection relationship between the library table nodes in the bloodline map. Fig.11 In the example, table C is generated by table A and table B through the task100, and table D is generated by table A and table C through the task100.

[0072] S130, uploading the blood relationship map and the index map to a preset map database and index database respectively.

[0073] Specifically, after constructing the bloodline map and index map of the task, the bloodline map and index map can be uploaded to the preset graph database and index data respectively through the bloodline reporting interface in the backend service. Among them, the bloodline map does not need to be uploaded through the bloodline reporting interface in the backend service, and it can be directly uploaded to the graph database through the channel between the bloodline data import layer and the graph database.

[0074] In other inventive embodiments, Fig.12 As shown, step S130 further includes steps S140, S150, S160 and S170.

[0075] S140, acquiring an upstream task for generating a database table to be analyzed from the graph database; wherein a task node of the upstream task is connected to a database table node of the database table to be analyzed;

[0076] S150, obtaining an index graph of the upstream task from the index database;

[0077] S160, obtaining a second running time for running the upstream task to generate the database table to be analyzed;

[0078] S170. Determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed according to the index map of the upstream task and the second running time.

[0079] In this embodiment, when it is necessary to determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed, it is necessary to take the database table to be analyzed as the starting point, obtain the upstream task connected to the database table to be analyzed from the lineage map of the graph database, and obtain the index map of the upstream task from the index map of the index database through the upstream task, and at the same time obtain the second running time when running the upstream task to generate the database table to be analyzed, and then the longest running time and the second running time for generating the upstream task can be obtained from the index map of the upstream task, so as to determine the most time-consuming path or the most resource-consuming path of the database table to be analyzed.

[0080] Specifically, when obtaining the upstream task of the database table to be analyzed, if the graph database is Nebula Graph, you can use the database table to be analyzed as the starting point and use the GO FROM statement or GET SUBGRAPH statement to obtain the upstream task of the database table to be analyzed from the NebulaGraph library. For example, if the database table to be analyzed is Figure 7 The D table in can be obtained from Figure 7 The upstream tasks are determined as task100, task101, and task102.

[0081] When obtaining the index graph of the upstream task from the index database, if the index database is the Nebula Graph library, you can also use the upstream task as the starting point and use the GO FROM statement or GET SUBGRAPH statement to obtain the index graph of the upstream task from the NebulaGraph library. Figure 7 For task100, task101, and task102 in the task list, the index map of the upstream task can refer to Fig.13 ,exist Fig.13 In the example, the downstream task of the upstream task task100 is taskX, the upstream tasks of the upstream task task100 are task98 and task99, the upstream task of the upstream task task98 is taskY, and the upstream task of the upstream task task101 is task98.

[0082] In addition, regarding the acquisition of the second running time, it is obtained from the log of calling the upstream task to generate the database table to be analyzed. The second running time of the upstream task that most recently ran to generate the database table to be analyzed is obtained by parsing the log. The second running time can be the average time of the upstream task that most recently ran multiple times to generate the database table to be analyzed.

[0083] In other inventive embodiments, Fig.14 As shown, step S170 includes steps S171 and S172.

[0084] S171, fusing the data to be analyzed, the second running time, and the index map of the upstream task to obtain a fused map;

[0085] S172. Using a critical path algorithm, determine from the fused graph the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed.

[0086] In this embodiment, in the process of determining the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed through the index map of the upstream task and the second running time, all the nodes of the downstream task of the upstream task of the data to be analyzed can be replaced with the nodes of the database table to be analyzed, and the connection relationship between the upstream task and the downstream task of the data to be analyzed can be replaced with the second running time, so that the data to be analyzed, the second running time and the index map of the upstream task can be fused, so as to obtain the following: Fig.15 The fused graph shown is finally processed by running the critical path algorithm to determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed.

[0087] In the lineage data collection method provided by the embodiment of the present invention, the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task are obtained; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship and input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, state and first running time of the predecessor task; the lineage information and the dependency relationship are written into the preset message middleware, and the lineage information and dependency relationship in the message middleware are processed to obtain the lineage map and index map of the real-time generated task; the lineage map and the index map are uploaded to the preset graph database and index database respectively. The present invention can not only realize the comprehensive collection of lineage data, but also can directly obtain the corresponding lineage map and index map from the graph database and index database when performing impact analysis on the data table in the lineage map, so that the longest time-consuming link or the most resource-consuming link of the entire task network upstream of the task where the data table is located can be tracked, and then the timeliness optimization and task resource optimization of the data table can be realized.

[0088] The embodiment of the present invention further provides a bloodline data collection device 100, which is used to execute any embodiment of the aforementioned bloodline data collection method.

[0089] Specifically, see Fig.16 , Fig.16 It is a schematic block diagram of the blood relationship data collection device 100 provided in an embodiment of the present invention.

[0090] like Fig.16 As shown, the blood relationship data collection device 100 includes: a first acquisition unit 110, a processing unit 120 and a first uploading unit 130.

[0091] The first acquisition unit 110 is used to acquire the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship and input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, state and first running time of the predecessor task;

[0092] The processing unit 120 is used to write the lineage information and the dependency relationship into a preset message middleware, and process the lineage information and dependency relationship in the message middleware to obtain a lineage graph and an index graph of the real-time generated task;

[0093] The first uploading unit 130 is used to upload the blood relationship map and the index map to a preset map database and index database respectively.

[0094] In other inventive embodiments, the first uploading unit 130 includes a second acquiring unit, a parsing unit, and a third acquiring unit.

[0095] A second acquisition unit, used for acquiring an SQL statement for generating the SQL task in real time and a first log for scheduling the SQL task in real time;

[0096] A parsing unit, used for parsing the SQL statement of the SQL task to obtain the lineage information of the SQL task;

[0097] The third acquisition unit is used to acquire the dependency relationship between the SQL task and its predecessor task from the relational database for real-time scheduling of the SQL task according to the first log.

[0098] In other inventive embodiments, the first uploading unit 130 includes a fourth acquiring unit and a fifth acquiring unit.

[0099] A fourth acquisition unit, used to acquire a second log for real-time scheduling of the non-SQL task;

[0100] A fifth acquisition unit is used to acquire, according to the second log, the lineage information of the non-SQL task and the dependency relationship between the non-SQL task and its predecessor task from a relational database for real-time scheduling of the non-SQL task.

[0101] In other inventive embodiments, the processing unit 120 includes a first building unit, a second building unit, a third building unit, a fourth building unit, a fifth building unit, and a sixth building unit.

[0102] A first construction unit is used to construct a plurality of library table nodes of the blood relationship map according to the data input library table and the data output library table;

[0103] A second construction unit is used to construct the bloodline map and a plurality of task nodes in the index map according to the identification information of the real-time generated task and the identification information of the predecessor task;

[0104] A third construction unit is used to construct a first connection relationship between a library table node and a task node in the blood relationship map according to the output relationship and the input relationship;

[0105] A fourth construction unit, configured to construct a second connection relationship between the task nodes in the index graph according to the dependency relationship;

[0106] A fifth construction unit, configured to construct the bloodline map according to the plurality of library table nodes, the plurality of task nodes and the first connection relationship of the bloodline map;

[0107] The sixth construction unit is used to construct the index map according to the multiple task nodes and the second connection relationship of the index map.

[0108] In other inventive embodiments, the fifth building block includes a sixth building block and a seventh building block.

[0109] A sixth construction unit, configured to construct a third connection relationship between the library table nodes in the bloodline graph according to the identification information of the real-time generation task;

[0110] The seventh construction unit is used to construct the bloodline map according to multiple library table nodes, multiple task nodes, the first connection key and the third connection relationship of the bloodline map.

[0111] In other inventive embodiments, the bloodline data collection device 100 further includes a sixth acquisition unit, a seventh acquisition unit and a first determination unit.

[0112] A sixth acquisition unit is used to acquire an upstream task for generating a database table to be analyzed from the graph database; wherein the task node of the upstream task is connected to the database table node of the database table to be analyzed;

[0113] A seventh acquisition unit, used to acquire the index graph of the upstream task from the index database;

[0114] An eighth acquiring unit, configured to acquire a second running time for running the upstream task to generate the database table to be analyzed;

[0115] The first determination unit is used to determine the longest time-consuming path or the most resource-consuming path of generating the database table to be analyzed according to the index map of the upstream task and the second running time.

[0116] In other inventive embodiments, the first determining unit includes: a fusion unit and a second determining unit.

[0117] A fusion unit, used to fuse the data to be analyzed, the second running time, and the index map of the upstream task to obtain a fused map;

[0118] The second determination unit is used to determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed from the fused graph by using a critical path algorithm.

[0119] The lineage data collection device 100 provided in the embodiment of the present invention is used to execute the above-mentioned acquisition of the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes the identification information of the real-time generated task, the data input library table, the data output library table, the output relationship and at least two of the input relationship, and the dependency relationship includes at least two of the identification information, status and first running time of the predecessor task; the lineage information and the dependency relationship are written into the preset message middleware, and the lineage information and dependency relationship in the message middleware are processed to obtain the lineage map and index map of the real-time generated task; the lineage map and the index map are uploaded to the preset graph database and index database respectively.

[0120] It should be noted that technical personnel in the relevant field can clearly understand that the specific implementation process of the above-mentioned bloodline data acquisition device 100 and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0121] The above-mentioned blood relationship data collection device can be implemented in the form of a computer program. The computer program can be used in Fig.17 Runs on the computer device shown.

[0122] See also Fig.17 , Fig.17 5 is a schematic block diagram of a computer device provided in an embodiment of the present application. The electronic device 500 may be a terminal, wherein the terminal may be a computer device with communication functions such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device.

[0123] See also Fig.17 The electronic device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .

[0124] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a bloodline data collection method.

[0125] The processor 502 is used to provide computing and control capabilities to support the operation of the entire electronic device 500 .

[0126] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a bloodline data collection method.

[0127] The network interface 505 is used to communicate with other devices over the network. Fig.17 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device 500 to which the scheme of the present application is applied. The specific electronic device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] Wherein, the processor 502 is used to run the computer program 5032 stored in the memory to implement the following steps: obtaining the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes the identification information, data input library table, data output library table, output relationship and at least two of the input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, status and first running time of the predecessor task; writing the lineage information and the dependency relationship into the preset message middleware, and processing the lineage information and dependency relationship in the message middleware to obtain the lineage map and index map of the real-time generated task; uploading the lineage map and the index map to the preset graph database and index database respectively.

[0129] In one embodiment, the real-time generated task includes an SQL task. When the processor 502 implements the acquisition of the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task, it specifically implements the following steps: obtaining the SQL statement for generating the SQL task in real time and the first log for scheduling the SQL task in real time; parsing the SQL statement of the SQL task to obtain the lineage information of the SQL task; and obtaining the dependency relationship between the SQL task and its predecessor task from the relational database for scheduling the SQL task in real time according to the first log.

[0130] In one embodiment, the real-time generated task includes a non-SQL task. When the processor 502 implements the acquisition of the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task, it specifically implements the following steps: obtaining a second log of real-time scheduling of the non-SQL task; and obtaining the lineage information of the non-SQL task and the dependency relationship between the non-SQL task and its predecessor task from a relational database that schedules the non-SQL task in real time according to the second log.

[0131] In one embodiment, when the processor 502 processes the lineage information and dependency relationships in the message middleware to obtain the lineage map and index map of the real-time generated task, it specifically implements the following steps: constructing multiple library table nodes of the lineage map according to the data input library table and the data output library table; constructing multiple task nodes in the lineage map and the index map according to the identification information of the real-time generated task and the identification information of the predecessor task; constructing a first connection relationship between the library table nodes and the task nodes in the lineage map according to the output relationship and the input relationship; constructing a second connection relationship between the task nodes in the index map according to the dependency relationship; constructing the lineage map according to the multiple library table nodes, the multiple task nodes and the first connection relationship of the lineage map; constructing the index map according to the multiple task nodes and the second connection relationship of the index map.

[0132] In one embodiment, when the processor 502 implements the construction of the bloodline map based on multiple library table nodes, multiple task nodes and the first connection relationship of the bloodline map, it specifically implements the following steps: constructing the third connection relationship between the library table nodes in the bloodline map based on the identification information of the real-time generated task; constructing the bloodline map based on multiple library table nodes, multiple task nodes, the first connection key and the third connection relationship of the bloodline map.

[0133] In one embodiment, after the processor 502 implements the uploading of the bloodline map and the index map to the preset graph database and the index database respectively, it specifically implements the following steps: obtaining an upstream task for generating a database table to be analyzed from the graph database; wherein the task node of the upstream task is connected to the library table node of the database table to be analyzed; obtaining the index map of the upstream task from the index database; obtaining a second running time for running the upstream task to generate the database table to be analyzed; and determining the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed based on the index map of the upstream task and the second running time.

[0134] In one embodiment, when the processor 502 determines the longest time-consuming path or the most resource-consuming path to generate the database table to be analyzed based on the index graph of the upstream task and the second running time, it specifically implements the following steps: fusing the data to be analyzed, the second running time, and the index graph of the upstream task to obtain a fused graph; and using a critical path algorithm to determine the longest time-consuming path or the most resource-consuming path to generate the database table to be analyzed from the fused graph.

[0135] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0136] It can be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0137] Therefore, the present invention also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by the processor, the processor executes the following steps: obtaining the lineage information of the real-time generation task and the dependency relationship between the real-time generation task and its predecessor task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship and input relationship of the real-time generation task, and the dependency relationship includes at least two of the identification information, state and first running time of the predecessor task; writing the lineage information and the dependency relationship into the preset message middleware, and processing the lineage information and dependency relationship in the message middleware to obtain the lineage map and index map of the real-time generation task; uploading the lineage map and the index map to the preset graph database and index database respectively.

[0138] In one embodiment, the real-time generated task includes an SQL task. When the processor executes the program instructions to obtain the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task, the processor specifically implements the following steps: obtaining the SQL statement for generating the SQL task in real time and the first log for scheduling the SQL task in real time; parsing the SQL statement of the SQL task to obtain the lineage information of the SQL task; and obtaining the dependency relationship between the SQL task and its predecessor task from the relational database for scheduling the SQL task in real time according to the first log.

[0139] In one embodiment, the real-time generated task includes a non-SQL task. When the processor executes the program instructions to obtain the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task, it specifically implements the following steps: obtaining a second log of real-time scheduling of the non-SQL task; and obtaining the lineage information of the non-SQL task and the dependency relationship between the non-SQL task and its predecessor task from a relational database for real-time scheduling of the non-SQL task according to the second log.

[0140] In one embodiment, when the processor executes the program instructions to implement the processing of the lineage information and dependency relationships in the message middleware and obtains the lineage map and index map of the real-time generated task, the following steps are specifically implemented: multiple library table nodes of the lineage map are constructed according to the data input library table and the data output library table; multiple task nodes in the lineage map and the index map are respectively constructed according to the identification information of the real-time generated task and the identification information of the predecessor task; a first connection relationship between the library table node and the task node in the lineage map is constructed according to the output relationship and the input relationship; a second connection relationship between the task nodes in the index map is constructed according to the dependency relationship; the lineage map is constructed according to the multiple library table nodes, the multiple task nodes and the first connection relationship of the lineage map; the index map is constructed according to the multiple task nodes and the second connection relationship of the index map.

[0141] In one embodiment, when the processor executes the program instructions to realize the construction of the bloodline map based on the multiple library table nodes, multiple task nodes and the first connection relationship of the bloodline map, it specifically implements the following steps: constructing the third connection relationship between the library table nodes in the bloodline map based on the identification information of the real-time generated task; constructing the bloodline map based on the multiple library table nodes, multiple task nodes, the first connection key and the third connection relationship of the bloodline map.

[0142] In one embodiment, after the processor executes the program instructions to upload the bloodline map and the index map to a preset graph database and an index database respectively, it specifically implements the following steps: obtaining an upstream task for generating a database table to be analyzed from the graph database; wherein the task node of the upstream task is connected to the library table node of the database table to be analyzed; obtaining the index map of the upstream task from the index database; obtaining a second running time for running the upstream task to generate the database table to be analyzed; and determining the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed based on the index map of the upstream task and the second running time.

[0143] In one embodiment, when the processor executes the program instructions to achieve the determination of the longest time-consuming path or the most resource-consuming path to generate the database table to be analyzed based on the index map of the upstream task and the second running time, the processor specifically implements the following steps: fusing the data to be analyzed, the second running time, and the index map of the upstream task to obtain a fused map; and using a critical path algorithm to determine the longest time-consuming path or the most resource-consuming path to generate the database table to be analyzed from the fused map.

[0144] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0145] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0146] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0147] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0148] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0149] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for collecting blood relationship data, characterized in that: include: Obtaining lineage information of a real-time generated task and a dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship, and input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, state, and first running time of the predecessor task; Writing the lineage information and the dependency relationship into a preset message middleware, and processing the lineage information and the dependency relationship in the message middleware to obtain a lineage graph and an index graph of the real-time generated task; The blood relationship map and the index map are uploaded to a preset map database and an index database respectively.

2. The method for collecting blood relationship data according to claim 1, characterized in that: The real-time generation task includes an SQL task; The step of obtaining the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task includes: Acquire a SQL statement for generating the SQL task in real time and a first log for scheduling the SQL task in real time; Parsing the SQL statement of the SQL task to obtain the lineage information of the SQL task; Obtain a dependency relationship between the SQL task and its predecessor task from a relational database for real-time scheduling of the SQL task according to the first log.

3. The method for collecting blood relationship data according to claim 1, characterized in that: The real-time generation tasks include non-SQL tasks; The step of obtaining the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task includes: Obtain a second log for real-time scheduling of the non-SQL task; According to the second log, the lineage information of the non-SQL task and the dependency relationship between the non-SQL task and its predecessor task are obtained from a relational database for real-time scheduling of the non-SQL task.

4. The method for collecting blood relationship data according to claim 1, characterized in that: The processing of the lineage information and dependency relationships in the message middleware to obtain a lineage graph and an index graph of the real-time generated task includes: Constructing a plurality of library table nodes of the blood relationship map according to the data input library table and the data output library table; Constructing the bloodline graph and a plurality of task nodes in the index graph respectively according to the identification information of the real-time generated task and the identification information of the predecessor task; Constructing a first connection relationship between a library table node and a task node in the bloodline graph according to the output relationship and the input relationship; Constructing a second connection relationship between task nodes in the index graph according to the dependency relationship; Constructing the bloodline map according to the plurality of library table nodes, the plurality of task nodes and the first connection relationship of the bloodline map; The index map is constructed according to the multiple task nodes and the second connection relationship of the index map.

5. The method for collecting blood relationship data according to claim 4, characterized in that: The step of constructing the bloodline map according to the plurality of library table nodes, the plurality of task nodes and the first connection relationship of the bloodline map includes: Constructing a third connection relationship between each library table node in the blood relationship map according to the identification information of the real-time generation task; The bloodline map is constructed according to a plurality of library table nodes, a plurality of task nodes, a first connection key and a third connection relationship of the bloodline map.

6. The method for collecting blood relationship data according to any one of claims 1 to 5, characterized in that: After uploading the bloodline map and the index map to the preset map database and index database respectively, the method further includes: Acquire an upstream task for generating a database table to be analyzed from the graph database; wherein a task node of the upstream task is connected to a database table node of the database table to be analyzed; Acquire the index graph of the upstream task from the index database; Acquire a second running time for running the upstream task to generate the database table to be analyzed; The longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed is determined according to the index map of the upstream task and the second running time.

7. The method for collecting blood relationship data according to claim 6, characterized in that: The determining, based on the index graph of the upstream task and the second running time, of generating the longest time-consuming path or the most resource-consuming path of the database table to be analyzed includes: Fusing the data to be analyzed, the second running time, and the index graph of the upstream task to obtain a fused graph; The critical path algorithm is used to determine the longest time-consuming path or the most resource-consuming path for generating the database table to be analyzed from the fused graph.

8. A blood relationship data collection device, characterized in that: include: A first acquisition unit is used to acquire the lineage information of the real-time generated task and the dependency relationship between the real-time generated task and its predecessor task; wherein the lineage information includes at least two of the identification information, data input library table, data output library table, output relationship and input relationship of the real-time generated task, and the dependency relationship includes at least two of the identification information, state and first running time of the predecessor task; A processing unit, used for writing the blood relationship information and the dependency relationship into a preset message middleware, and processing the blood relationship information and the dependency relationship in the message middleware to obtain a blood relationship map and an index map of the real-time generated task; The first uploading unit is used to upload the blood relationship map and the index map to a preset map database and an index database respectively.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the bloodline data collection method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to perform the bloodline data collection method according to any one of claims 1 to 7.

Images