Modeling method and system based on DolphinScheduler task instance monitoring and dependency relationship

Abstract

The invention discloses a DolphinScheduler task instance monitoring and dependency relationship modeling method and system, and the method comprises the following steps: monitoring an instance change event flow of a DolphinScheduler, and obtaining a change event carrying a task identifier; associating task definition data in a memory cache based on the task identifier to obtain a context object containing a task type and a parameter; performing dependency analysis on the SQL type task, and obtaining an input table list depended by the SQL task and a generated output table list based on the task parameters in the context object; according to the context object, the input table list and the output table list, constructing a task definition node, a task instance node, a table node and a logic relation edge among the three; and writing the nodes and the edges into a graph database to form a dependency relationship model. The method can automatically derive upstream and downstream dependency chains between tasks based on the table-level data production consumption relationship, and breaks through the limitation that traditional dependency can only be based on projects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing and task scheduling, and particularly relates to a method and system for task instance monitoring and dependency relationship modeling based on DolphinScheduler. BACKGROUND

[0002] In the prior art, a distributed task scheduling system, DolphinScheduler scheduling system, is widely used to organize and arrange multiple types of tasks (such as SQL scripts, Shell scripts, Python scripts, etc.). However, it still has the following problems: 1. The execution state and dependency relationship of the task instance lack intuitiveness, and it is difficult to quickly locate the fault node in the complex task chain.

[0003] 2. The traditional dependency relationship analysis is only based on the preset upstream and downstream relationship of the scheduling system, and does not combine the table-level dependency relationship in the actual execution SQL statement of the task, resulting in an incomplete dependency graph.

[0004] 3. The task dependency is configured manually, and the dependency changes caused by the SQL logic changes cannot be dynamically captured, and the tasks between projects cannot establish dependency relationships.

[0005] 4. There is a lack of real-time monitoring and graph construction capability based on task execution data (such as execution time and state). SUMMARY

[0006] According to the embodiment of the present application, a method for task instance monitoring and dependency relationship modeling based on DolphinScheduler is provided, which comprises the following steps: Monitoring the instance change event stream of DolphinScheduler to obtain a change event carrying a task identifier; Associating the task definition data in the memory cache based on the task identifier to obtain a context object containing the task type and parameters; Performing dependency analysis on the SQL type task to obtain the input table list and the generated output table list of the SQL task dependency based on the task parameters in the context object; According to the context object, the input table list and the output table list, constructing task definition nodes, task instance nodes, table nodes and the logical relationship edges among them; Writing the nodes and edges into a graph database to form a dependency relationship model.

[0007] Further, monitoring the instance change event stream of DolphinScheduler comprises the following steps: Using Debezium to monitor the task instance table of DolpinScheduler to capture the change event of the task instance; Convert the change event into a standard data structure and push to message middleware for transmission.

[0008] Further, the message middleware uses Kafka.

[0009] Further, based on the task identifier, the task definition data in the memory cache is associated to obtain a context object containing task type and parameters, including the following steps: Load the task definition data into the memory cache; Query the memory key-value cache with the task identifier as the key to extract the corresponding task definition information; Fusion task instance event and task definition information to generate context object.

[0010] Further, for SQL type tasks, the input table list and generated output table list of the SQL task are obtained based on the task parameters in the context object, including the following steps: Identify the task type in the context object as a SQL task and extract the SQL text; Parse the SQL text to generate an abstract syntax tree; Traverse the abstract syntax tree to extract the input tables in the FROM / JOIN clause; Traverse the abstract syntax tree to extract the output tables in the INSERT INTO / CREATE TABLE AS clause.

[0011] Further, the following steps are also included: stripping the catalog prefix of the input tables and output tables and converting them to the standard naming format of database.table.

[0012] Further, the logical relationship between the three includes the execution edge from the task instance node to the task definition node, the input edge from the table node to the task definition node, and the output edge from the task definition node to the table node.

[0013] Further, writing the nodes and edges into the graph database to form a dependency relationship model includes the following steps: Create a unique index for the task identifier of the task definition node and the table name of the table node; Update when the node exists, or insert a new node when it does not exist; Update when the edge exists, or insert a new edge when it does not exist; Periodically clean up isolated nodes and expired edges scanned; Write the nodes and edges into the graph database to form a dependency relationship model.

[0014] Further, the graph database uses NebulaGraph graph database.

[0015] According to another embodiment of the present application, a system for DolphinScheduler task instance monitoring and dependency modeling is provided, comprising: An event monitoring module, the event monitoring module monitors an instance change event stream of the DolphinScheduler, and obtains a change event carrying a task identifier; A completion module, the completion module associates task definition data in a memory cache based on the task identifier, and obtains a context object containing a task type and parameters; An analysis module, the analysis module analyzes a dependency relationship of the SQL task, and obtains an input table list and an output table list generated by the SQL task based on the task parameters in the context object; A construction module, the construction module constructs a task definition node, a task instance node, a table node, and a logical relationship edge among the three based on the context object, the input table list, and the output table list, and writes the nodes and the edge into a graph database to form a dependency relationship model.

[0016] The method and system for DolphinScheduler task instance monitoring and dependency modeling according to the embodiments of the present application automatically deduce an upstream and downstream dependency chain among tasks based on table-level data production and consumption relationships, break through the limitation that traditional dependencies can only be based on projects, support quick positioning of a full-link dependency relationship of a fault node through graph traversal, display a data flow path of a complex task chain, and analyze a dependency chain performance bottleneck in combination with execution time consumption, thereby providing a full-process intelligent management scheme of "data collection-dependency modeling-real-time monitoring-fault positioning" for a large-scale task scheduling system in production, and significantly improving operation and maintenance efficiency and system stability.

[0017] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the subject technology. BRIEF DESCRIPTION OF DRAWINGS

[0018] Fig. 1 FIG. 1 is a schematic diagram of a method for DolphinScheduler task instance monitoring and dependency modeling according to an embodiment of the present application; Fig. 2 FIG. 2 is a flowchart of a method for DolphinScheduler task instance monitoring and dependency modeling according to an embodiment of the present application; Fig. 3 FIG. 3 is a flowchart of monitoring an instance change event stream of the DolphinScheduler according to a method for DolphinScheduler task instance monitoring and dependency modeling according to an embodiment of the present application; Fig. 4A flowchart for obtaining task definition data in a memory cache based on a task identifier according to the method for monitoring and modeling a dependency relationship of a task instance based on DolphinScheduler according to an embodiment of the application, obtaining a context object containing a task type and parameters; Fig. 5 A flowchart for obtaining a list of input tables and a list of generated output tables of a SQL task dependency based on a task parameter in a context object according to the method for monitoring and modeling a dependency relationship of a task instance based on DolphinScheduler according to an embodiment of the application; Fig. 6 A flowchart for writing a node and an edge into a graph database to form a dependency relationship model according to the method for monitoring and modeling a dependency relationship of a task instance based on DolphinScheduler according to an embodiment of the application. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present application will be described in detail with reference to the drawings, and the present application will be further described.

[0020] First, the preferred embodiments of the present application will be described with reference to the drawings. Figs. 1-6 The method and system for monitoring and modeling a dependency relationship of a task instance based on DolphinScheduler according to an embodiment of the application are described for realizing visual monitoring of a task execution state and a dynamic dependency relationship and improving fault positioning efficiency of a complex task chain, and the application scenarios are very wide.

[0021] As shown in Figs. 1-6 The method for monitoring and modeling a dependency relationship of a task instance based on DolphinScheduler according to an embodiment of the application comprises the following steps: Step one: Monitor the instance change event stream of DolphinScheduler to obtain change events carrying task identifiers; Specifically, the task instance table t_ds_task_instance of DolphinScheduler is monitored, that is, the core data source is the table t_ds_task_instance in the DolphinScheduler database that records the details of each task running, Debezium is used to deploy the Binlog log collection component, the system can non-invasively and real-time listen to database change events, capture incremental change information of task instances, and whenever DolphinScheduler scheduling system creates, executes or updates task instances, the table will record relevant execution information, the core fields include: instance primary key (id), associated task definition number (task_code), current state (such as SUCCESS, FAILURE, etc.), start time (start_time), end time (end_time), retry times (retry_times) and the like; The system converts these change events into a standard data structure and pushes them to Kafka as the trigger source for subsequent modeling.

[0022] Step two: Associate the task definition data in the memory cache based on the task identifier to obtain a context object containing task type and parameters; Further, based on the task identifier, the task definition data in the memory cache is associated to obtain a context object containing task type and parameters, including the following steps: loading task definition data into memory cache; Query the memory key-value cache with the task identifier as the key to extract the corresponding task definition information; Fusion task instance event and task definition information to generate context object.

[0023] Specifically, since the t_ds_task_definition table data is relatively static, if the database query pressure will be very large if the database is queried every time to obtain the definition information, repeated reading of the database will produce performance bottleneck. Therefore, the system will load the contents of the t_ds_task_definition table into the memory cache in the initialization stage, and the cache structure is key-value pair: key (unique identifier of task definition task_code), Value (complete task definition object containing task type, SQL content, project ownership, task name, creator, etc.). The default setting is 10 minutes of expiration time (configurable), and supports event-driven refresh mechanism. If the system detects that the task definition has been modified (such as script content update, parameter change, etc.), the corresponding cache item is automatically refreshed to ensure the efficiency of SQL parsing process and the consistency of task metadata.

[0024] The downstream service continuously reads the event stream of task instances from Kafka. Since the task instance event itself only contains the task_code and related identifiers such as task execution time, and does not directly contain complete task definition information, the system needs to complete the information such as the task belonging to flow, project and the like from the memory cache. The completed context will be used for subsequent steps such as SQL parsing, graph node creation and the like.

[0025] Step three: dependency parsing of the SQL type task, obtaining the input table list and the generated output table list of the SQL task dependency based on the task parameters in the context object; further, the dependency parsing of the SQL type task, obtaining the input table list and the generated output table list of the SQL task dependency based on the task parameters in the context object includes the following steps: identifying the task type in the context object as a SQL task, extracting the SQL text; parsing the SQL text to generate an abstract syntax tree; traversing the abstract syntax tree to extract the input table in the FROM / JOIN clause; traversing the abstract syntax tree to extract the output table in the INSERT INTO / CREATE TABLE AS clause.

[0026] Further, it further includes the following steps: stripping the catalog prefix of the input table and the output table, and converting it into the standard naming format of database.table.

[0027] Specifically, for the task of type SQL, the system parses the sql field in task_params in the instance running json stored in kafka to obtain its SQL text for dependency parsing, and the specific process includes: Parsing tool: use SQL parsing engine to convert SQL to AST (abstract syntax tree).

[0028] Input table identification: locate the table in FROM and JOIN.

[0029] Output table identification: parse the target table of INSERT INTO and CREATE TABLE AS.

[0030] Table name standardization: stripping the catelog prefix and unifying the database.table naming format.

[0031] Step four: constructing task definition nodes, task instance nodes, table nodes and logical relationship edges among the three according to the context object, the input table list and the output table list; further, the logical relationship edges among the three include the execution edge from the task instance node to the task definition node, the input edge from the table node to the task definition node, and the output edge from the task definition node to the table node.

[0032] Specifically, Vertex node types: 1) Task Definition Vertex: represents static task definitions such as task_code, task_name, flow_name, flow_code, project_name, project_code, etc.

[0033] 2) Task Instance Vertex: represents a task running instance, including state, start_time, submit_time, task_type, etc.

[0034] 3) Table Vertex: represents data table names, supporting multiple tasks sharing the same table node.

[0035] Edge types: 1) Executed By Edge: Task Instance Vertex→Task Definition Vert, indicating which task definition generates the instance.

[0036] 2) From Edge: Table Vertex→Task Definition Vert, indicating which input tables the task definition depends on.

[0037] 3) To Edge: Task Definition Vertex→Table Vert, indicating which output tables the task definition writes to.

[0038] Step five: write nodes and edges into the graph database to form a dependency relationship model. Further, writing nodes and edges into the graph database to form a dependency relationship model includes the following steps: create a unique index for the task identifier of the task definition node and the table name of the table node; update when the node exists, insert a new node when it does not exist; update when the edge exists, insert a new edge when it does not exist; periodically clean up scanned isolated nodes and expired edges; write nodes and edges into the graph database to form a dependency relationship model.

[0039] Specifically, the constructed graph data structure will be written into the graph database (NebulaGraph graph database is used in this invention) to achieve efficient dependency relationship storage and query. The writing logic is as follows: a. Check for duplicates, if the node or edge already exists, update it; b. Add new, if it does not exist, create a new node and edge; c. Set unique indexes for task_code, qualified_table_name, etc. to ensure consistency; d. Periodically clean up scanned isolated nodes and expired edges to keep the graph structure compact.

[0040] In summary, by listening to the Binlog data of the t_ds_task_instance table of DolphinScheduler, the full life cycle state of the task instance is captured in real time, without relying on periodic database queries, which greatly reduces the monitoring delay. By parsing the task SQL statement, the implicit table-level dependency (such as the input-output relationship associated by the table name in the SQL) is mined, and the automatic construction of the task dependency relationship across projects is supported without manual configuration. When the task SQL logic changes (such as adding table association), the dependency relationship is automatically updated, solving the problem that the traditional static dependency configuration cannot cope with business logic changes According to another embodiment of the application, a system for monitoring task instances and modeling dependencies based on DolphinScheduler is provided, comprising: an event monitoring module, the event monitoring module monitors the instance change event stream of DolphinScheduler, and obtains a change event carrying a task identifier; a completion module, the completion module associates the task definition data in the memory cache based on the task identifier, and obtains a context object containing the task type and parameters; an analysis module, the analysis module analyzes the dependency relationship of the SQL task, and obtains the input table list and the generated output table list of the SQL task dependency based on the task parameters in the context object; a construction module, the construction module constructs the task definition node, the task instance node, the table node and the logical relationship edge among them according to the context object, the input table list and the output table list, and writes the nodes and edges into a graph database to form a dependency relationship model.

[0041] The above, with reference to Figs. 1-6 The method and system for monitoring task instances and modeling dependencies based on DolphinScheduler according to the embodiments of the application are described, the upstream and downstream dependency chain among tasks is automatically derived based on the table-level data production and consumption relationship, breaking through the limitation that the traditional dependency can only be based on projects. The full-link dependency relationship of the fault node is quickly located by graph traversal, the data flow path of the complex task chain is displayed, and the performance bottleneck of the dependency chain is analyzed in combination with the execution time, providing a full-process intelligent management scheme of "data collection-dependency modeling-real-time monitoring-fault location" for large-scale task scheduling systems in production, significantly improving the operation and maintenance efficiency and system stability.

[0042] It should be noted that, in this specification, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0043] While the application has been described in detail by reference to preferred embodiments thereof, it should be understood that many modifications and substitutions can be made by those skilled in the art without departing from the spirit and scope of the present application, as set forth in the following claims.

Claims

1. A method for modeling task instance monitoring and dependency relationship based on DolphinScheduler, characterized in that, Comprising the following steps: Monitor the instance change event stream of DolphinScheduler to obtain a change event carrying a task identifier; Based on the task identifier, associate the task definition data in the memory cache to obtain a context object containing task type and parameters; Parse the dependencies of SQL type tasks, and based on the task parameters in the context object, obtain the input table list and generated output table list that the SQL task depends on; According to the context object, the input table list and the output table list, construct task definition nodes, task instance nodes, table nodes and the logical relationship edges among them; Write the nodes and edges into a graph database to form a dependency relationship model.

2. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 1, characterized in that, The monitoring of the instance change event stream of DolphinScheduler comprises the following steps: Use Debezium to monitor the task instance table of DolpinScheduler to capture the change event of the task instance; Convert the change event into a standard data structure and push it to the message middleware for transmission.

3. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 2, characterized in that, The message middleware uses Kafka.

4. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 1, characterized in that, The association of the task identifier with the task definition data in the memory cache to obtain a context object containing task type and parameters comprises the following steps: Load the task definition data into the memory cache; Query the memory key-value cache with the task identifier as the key to extract the corresponding task definition information; Fuse the task instance event and the task definition information to generate a context object.

5. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 1, characterized in that, The dependency parsing of SQL type tasks based on the task parameters in the context object to obtain the input table list and the generated output table list that the SQL task depends on comprises the following steps: Identify the task type in the context object as a SQL task and extract the SQL text; Parse the SQL text to generate an abstract syntax tree; Traverse the abstract syntax tree to extract the input tables in the FROM / JOIN clause; Traverse the abstract syntax tree to extract the output tables in the INSERT INTO / CREATE TABLE AS clause.

6. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 5, wherein, Further comprising the following steps: strip the catalog prefix of the input tables and the output tables and convert them to the standard naming format of database.table.

7. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 1, characterized in that, The logical relationship edges among the three comprise execution edges from task instance nodes to task definition nodes, input edges from table nodes to task definition nodes, and output edges from task definition nodes to table nodes.

8. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 1, characterized in that, The writing of nodes and edges into a graph database to form a dependency relationship model comprises the following steps: Create unique indexes for the task identifier of the task definition node and the table name of the table node; Update when the node exists, or insert a new node when it does not exist; Update when the edge exists, or insert a new edge when it does not exist; Periodically clean up isolated nodes and expired edges; Write the nodes and edges into a graph database to form a dependency relationship model.

9. The method for modeling task instance monitoring and dependency relationship based on DolphinScheduler according to claim 1 or 8, characterized in that, The graph database uses NebulaGraph graph database.

10. A system for modeling task instance monitoring and dependency relationship based on DolphinScheduler, characterized in that, Comprise: An event monitoring module, the event monitoring module monitors the instance change event stream of DolphinScheduler to obtain a change event carrying a task identifier; The supplement module obtains the context object containing the task type and the parameters based on the task identifier associated with the task definition data in the memory cache; The analysis module analyzes the dependency relationship of the SQL task, and obtains the input table list and the generated output table list of the SQL task based on the task parameters in the context object; The construction module constructs the task definition node, the task instance node, the table node and the logical relationship edge among the three according to the context object, the input table list and the output table list, and writes the node and the edge into the graph database to form the dependency relationship model.

Images