DAG-based task flow execution method and device, equipment and storage medium

Abstract

The invention provides a DAG-based task flow execution method, which comprises the following steps that: a DAG task flow to be executed at present is obtained from pre-stored DAG task data, and the DAG task flow is stored in the DAG task data in the form of a structure class diagram; extracting task nodes and an execution sequence among the task nodes according to the DAG task flow; each task node is used for representing a task type; based on the task type, searching a plug-in corresponding to each task node from a pre-constructed plug-in library; and executing the plug-ins in sequence according to the execution sequence, ending the execution of the plug-ins in sequence, and ending the execution of the DAG task flow. According to the method, the constructed DAG task flow is stored in the form of the structure class diagram, when a new data source or task type needs to be expanded, the information of the new task type can be directly classified and stored in the structure class diagram, and the expandability of a DAG task scheduling system can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of task flow technology, and in particular to a task flow execution method, apparatus, device and storage medium based on DAG. Background Technology

[0002] In today's era of big data and cloud computing, data acquisition is the cornerstone of data analysis, business intelligence, and artificial intelligence applications. As the "central nervous system" of the data pipeline, the acquisition scheduling platform is primarily responsible for managing, scheduling, and monitoring numerous and complex data acquisition tasks. Its core mission is to ensure that tasks are executed in an orderly and efficient manner according to the correct dependencies. Currently, task scheduling systems based on DAG (Directed Acyclic Graph) are commonly used to implement acquisition scheduling platforms. These systems typically employ a DAG task flow execution method based on a database and a centralized scheduler. The core idea of ​​this method is to persist the DAG structure and have a central scheduler handle global resource allocation and task dispatch. In other words, traditional scheduling execution methods store the constructed DAG task flow in a structured manner (e.g., storing the DAG as a fixed table). When executing DAG tasks, the pre-deployed parsing logic based on the current structured DAG task flow is loaded and executed from the database. This method requires modification of the database and parsing logic, and may even require code redeployment, when expanding to new data sources or task types, resulting in poor scalability.

[0003] Therefore, how to improve the scalability of DAG task scheduling systems has become an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, this application proposes a task flow execution method, apparatus, device and storage medium based on DAG, which can effectively improve the scalability of DAG task scheduling system.

[0005] According to a first aspect of this application, a task flow execution method based on DAG is provided, comprising: The DAG task flow to be executed is obtained from the pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structure class diagram; Based on the DAG task flow, each task node and the execution order between each task node are extracted; each task node is used to represent a task type. Based on the task type, the plugin corresponding to each task node is searched from a pre-built plugin library; Each plugin is executed sequentially according to the execution order, and the DAG task flow ends when each plugin finishes execution.

[0006] In one possible implementation, obtaining the execution order among the task nodes includes: The dependency relationships between the task nodes are constructed based on the DAG task flow; The execution order of each task node is determined based on the dependencies.

[0007] In one possible implementation, the execution order includes at least one of parallel execution and sequential execution.

[0008] In one possible implementation, the plugin contains execution logic for the corresponding task type.

[0009] In one possible implementation, when executing each of the plugins sequentially based on the execution order, the following is included: Based on the execution order, obtain the plugin corresponding to the current task node, and execute the plugin to obtain the execution result; Based on the execution result, determine whether to continue executing the next task node. If the execution result is determined to be successful, the plugin corresponding to the next task node will continue to be executed.

[0010] In one possible implementation, when the DAG task flow is stored in the DAG task data in the form of a structural class graph, the structural class graph includes at least one of the following: graph class, node class, edge information class, node data class, node port class, edge source information class, and edge target information class.

[0011] In one possible implementation, the DAG task flow is stored in the form of a structural class graph before the DAG task data, and the implementation also includes the operation of constructing the DAG task flow.

[0012] According to a second aspect of this application, a DAG-based task flow execution apparatus is provided, comprising: The DAG task flow acquisition module is used to acquire the DAG task flow to be executed from the pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structure class diagram; The DAG task flow data extraction module is used to extract each task node and the execution order between each task node according to the DAG task flow; each task node is used to represent a task type. The plugin search module is used to search for the plugins corresponding to each task node from a pre-built plugin library based on the task type. The plugin execution module is used to execute each plugin sequentially according to the execution order. Once each plugin has finished executing, the DAG task flow execution ends.

[0013] According to a third aspect of this application, a DAG-based task flow execution device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method described in the first aspect of this application.

[0014] According to a fourth aspect of this application, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the method described in the first aspect of this application.

[0015] This application provides a DAG-based task flow execution method, comprising: obtaining the DAG task flow to be executed from pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structural class diagram; extracting each task node and the execution order between the task nodes according to the DAG task flow; each task node representing a task type; searching for the corresponding plugin for each task node from a pre-built plugin library based on the task type; executing each plugin sequentially according to the execution order, and the DAG task flow execution ends when each plugin finishes execution sequentially. This application packages each task type as a plugin, so that each task type's corresponding plugin can be called for independent execution during execution. When it is necessary to expand to new data sources or task types, it is only necessary to package the new data source or task type as a corresponding plugin and store it in the plugin library, and then the new task type can be flexibly added to the original DAG task flow. Through the plugin-based setting, the corresponding plugin can be directly called for execution during task scheduling, without redeploying the parsing logic, thereby effectively improving the scalability of the DAG task scheduling system. Furthermore, this application stores the constructed DAG task flow in the form of a structural class diagram. Thus, after registering a new task type, when the DAG task flow changes, only the new task node information needs to be stored according to the class, without redeploying the database, which improves the scalability of the DAG task scheduling system.

[0016] Other features and aspects of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0017] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0018] Figure 1 A flowchart is shown showing a DAG-based task flow execution method according to an embodiment of this application; Figure 2This diagram illustrates the plugin deployment in a DAG-based task flow execution method according to an embodiment of the present application. Figure 3 This diagram illustrates a DAG-based task flow storage structure according to an embodiment of the present application. Figure 4 A JSON data structure diagram of a DAG-based task flow according to an embodiment of this application is shown; Figure 5 This illustrates a class diagram of a class loader in a DAG-based task flow according to an embodiment of this application; Figure 6 This illustrates a DAG-based task flow execution class diagram according to an embodiment of this application; Figure 7 A flowchart illustrating a DAG-based task flow execution method according to an embodiment of this application is shown. Figure 8 This diagram illustrates the loading of a DAG-based task flow execution plugin class loader according to an embodiment of this application. Figure 9 This diagram shows a schematic block diagram of a DAG-based task flow execution apparatus according to an embodiment of the present application; Figure 10 A schematic block diagram of a DAG-based task flow execution device according to an embodiment of this application is shown. Detailed Implementation

[0019] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0020] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0021] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0022] <Method Implementation> Figure 1 A flowchart illustrating a DAG-based task flow execution method according to an embodiment of this application is shown. Figure 1As shown, the method includes steps S1100-S1400: S1100, obtaining the DAG task flow to be executed from the pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structural class diagram; S1200, extracting each task node and the execution order between each task node according to the DAG task flow; each task node is used to represent a task type; S1300, searching for the corresponding plugin for each task node from the pre-built plugin library based on the task type; S1400, executing each plugin sequentially according to the execution order, and the execution of the DAG task flow ends when each plugin finishes executing sequentially.

[0023] This application encapsulates the execution logic corresponding to each task type into independent plugins, allowing each task to run independently during execution. Furthermore, by encapsulating each task type into independent plugins, when expanding to include new data sources or task types, the new data source or task type can be directly packaged into a corresponding new plugin and stored in a plugin library. After registering the task type corresponding to the new plugin in the plugin library, it can be arbitrarily added to the existing DAG task flow for execution. This process only requires registering the task type corresponding to the new plugin, eliminating the need for complex operations and thus improving the scalability of the DAG task scheduling system. Further, this application stores the constructed DAG task flow in the form of a structural class diagram. When adding a new data source or task type to the task flow, the various data types of the newly added task type can be directly added to the corresponding classes in the structural class diagram without redeploying the database, effectively improving the scalability of the DAG task scheduling system.

[0024] It should be noted that when retrieving the DAG task flow to be executed from the pre-stored DAG task data, the pre-constructed DAG task flows are first stored as DAG task data, and then the DAG task flow to be executed is retrieved from the DAG task data. It should also be noted that the DAG task data contains multiple DAG task flows, each stored in the form of a structured class diagram. Prior to storing the DAG task flows in the form of structured class diagrams within the DAG task data, the process also includes the operation of constructing the DAG task flows.

[0025] In one possible implementation, constructing the DAG task flow includes: obtaining the task types corresponding to each plugin in a pre-built plugin library; registering each task type and rendering each registered task type as a corresponding task node on the front end; and constructing the DAG task flow based on the task nodes rendered on the front end. Specifically, when pre-building the plugin library, the execution logic and dependencies corresponding to each task type are packaged into plugins and stored in the database to obtain the pre-built plugin library. It should be noted that the execution dependencies of each task type refer to the classes called when executing the corresponding execution logic. It should also be noted that the database has hot deployment interfaces including at least one of addJar() and close(), where addJar() is used to dynamically add new plugin JARs and close() is used to unload plugins and release resources.

[0026] In one possible implementation, see [link to relevant documentation] Figure 2 As shown, when packaging the execution logic and dependencies corresponding to each task type into plugins, all plugins uploaded by developers (including JARs) must be released in "All-in-One JAR" format, including: (1) Plugin main class (implementing PluginInterface) (2) lib / directory: All third-party dependency JARs (such as commons-net-3.8.0.jar for FTP client) (3) plugin-meta.properties: Plugin metadata (basic properties such as task type, name, description, icon, version, author, dependency tree validation) The internal structure of the plugin is shown below: ftp-plugin.jar ├── {prefix} / ftp / FtpIngestPlugin.class # Main plugin class ├── lib / │ ├── commons-net-3.8.0.jar │ └── jsch-0.1.55.jar └── plugin-meta.properties When packaging the execution logic and dependencies for each task type into plugins, a dependency conflict redirection strategy is also implemented. That is, when version conflicts occur while packaging third-party dependencies into plugins, the dependency is directly copied to the path of the refactored dependency and repackaged. In other words, the third-party dependency can be directly copied to the dependency path within the plugin. This can be represented by the following executable code: <!-- Plugin construction configuration (patent feature) --> <plugin> <groupid> org.apache.maven.plugins< / groupid> <artifactid> maven-shade-plugin< / artifactid> <executions> <execution> <phase> package< / phase> <goals> <goal> shade< / goal> < / goals> <configuration> <relocations> <!-- Redirect easily conflicting dependencies --> <relocation> <pattern> org.apache.commons< / pattern> <shadedpattern> com.plugin.shaded.commons< / shadedpattern> < / relocation> < / relocations> < / configuration> < / execution> < / executions> < / plugin> See Figure 2 As shown, the developer uploads the plugins corresponding to each task type that have been packaged, and stores each plugin in the plugin library (that is, Figure 2 in the plugin repository). After that, since there is a plugin interface deployed on the executor for interacting with the plugin library. That is, the executor scans each plugin in the plugin library based on this plugin interface, and creates a registration instance for each plugin (that is, Figure 2 in, create a PluginClassLoader instance), and then verifies the dependency signatures (such as SHA-256) in each plugin. If the verification passes, the task types corresponding to each plugin (such as, type=ftp_ingest) are registered with the scheduling center. Among them, the plugin interface usually includes two core methods: execute(Context context) and getSupportedType(). execute(Context context) is used to obtain the execution method, so as to be called when executing the execution logic of the plugins corresponding to each task node. That is, when executing each task node, the execute(Context context) method of the plugin interface can be used to directly call the execution logic of the plugin corresponding to the current task node for execution. getSupportedType() is used by the user to obtain the task types supported by the plugin, and is used when registering the task types corresponding to the plugin. That is, when registering each task type with the scheduling center, the getSupportedType() method of the plugin interface of the executor scans the plugins related to each task class in the plugin library, creates instances corresponding to each task type, completes the registration task of each task type in the scheduling center, and renders each task type into corresponding task nodes on the front-end display interface.

[0027] Among them, when registering each task type with the scheduler, the execution code can be as follows: / / Create a plugin class loader URL[] jarUrls = {new URL("file: / / / plugins / ftp-plugin.jar")}; PluginClassLoader loader = new PluginClassLoader(jarUrls,parentClassLoader); / / Load the main class of the plugin Class<?> pluginClass = loader.loadClass("{prefix}.ftp.FtpIngestPlugin"); PluginInterface plugin = (PluginInterface) pluginClass.newInstance(); / / Register the task types supported by the plugin with the scheduler scheduler.registerPlugin(plugin.getSupportedType(), plugin); It should also be noted that a plugin directory (such as / plugins) is deployed in the executor, and the executor is set to incremental scan time. This allows the plugins corresponding to newly added task types in the plugin repository to be identified by comparing the plugin directory. Then, when a new task type plugin is added, the new task type can be registered to the scheduling center in real time according to the above registration operation.

[0028] After registering each task type with the scheduling center, this application sets up a visual designer to obtain the registered task types and their metadata from the scheduling center via API. This allows the visual designer to render and display each task type on the front-end interface, making each task type a node component within the task node component set. When building a task flow, users can directly drag and drop the required task node components from the component library displayed on the visual designer's front-end interface as task nodes in the constructed task flow. Then, based on actual needs, the task nodes are connected with directed line segments to construct a DAG task flow.

[0029] It's important to note that task types are typically developed and registered with the scheduling center as plugins. Each plugin provides its own metadata during registration to facilitate management, version control, and dependency management. The metadata for each task type includes at least one of the following: basic attributes (such as task type, name, description, and icon), plugin version (for compatibility checks), author (for attribution identification), and dependency tree verification (for verifying dependency integrity). After each task type is registered with the scheduling center, the center pushes the task type and its metadata to the visual designer via API. This allows the front-end to obtain complete task type information, enabling users to fully understand the task type attributes when building DAG task flows, ensuring the reliability and maintainability of the task flow.

[0030] When rendering the corresponding task node components for each task type on the front end, this metadata is mainly used for the following purposes: Core display information (display name, description, icon): Display task nodes in the component library so that users can identify and select them.

[0031] Version information: This displays the version number of task nodes in the front-end interface, helping users identify different versions of task types and avoid using outdated or incompatible components. When building a DAG task flow, version information can be used to prompt for upgrades or check version consistency between nodes.

[0032] Author information: Displays the developer or source of the task node, increasing traceability. Users can view the author on the front end for easy feedback or support.

[0033] Dependency Validation: When users connect task nodes to build a DAG, the front end can use dependency tree validation and quick verification to check whether the dependencies between task nodes are valid (e.g., check for circular dependencies or missing dependencies).

[0034] Integrity check: When dragging nodes or saving task flow, the front end can perform a check on the integrity of the dependency tree based on validation and pre-check to prevent task flow errors caused by missing dependencies.

[0035] Visual hints: The front end may render nodes based on dependency status (such as color coding). For example, if a dependency is not satisfied, the node may be displayed in a warning color to guide the user to make adjustments.

[0036] The scheduling center enhances the interactivity and reliability of the front end by pushing metadata of each task type to the front end for rendering and display. This allows users to receive real-time feedback when dragging and dropping to build DAG task flows, reducing the risk of errors.

[0037] Based on the above description, when constructing a DAG task flow, users drag and drop each task node component onto the construction interface based on the information displayed therein. Each task node and edge is uniquely identified by an ID. The construction interface also has coordinates, which can be used to describe each task node. When connecting task nodes with directed line segments, the port and port group of the task node being connected must be clearly defined. This allows for the direct storage of various information according to their respective categories in the class diagram when storing the constructed DAG task flow as a structural class diagram.

[0038] For example, the constructed DAG task flow can be represented using a JSON data structure as follows: { "nodes": [ { "id": "node1", "shape": "dag-node", "data": {"label": "FTP collection", "type": "ftp-ingest"}, "ports": [{"id": "out1", "group": "bottom"}] }, { "id": "node2", "shape": "dag-node", "data": {"label": "Data Cleaning", "type": "data-clean"}, "ports": [{"id": "in1", "group": "top"}, {"id": "out2", "group": "bottom"}] } ], "edges": [ { "id": "edge1", "shape": "dag-edge", "source": {"cell": "node1", "port": "out1"}, "target": {"cell": "node2", "port": "in1"} } ] } The incremental scan time can be preferably set to scan every 30 seconds. When a new task type is detected, the user can modify the constructed DAG task flow based on the actual situation (i.e., add a new task type node to the original DAG task flow). In other words, the visual designer (used to display the visual task flow construction interface) automatically retrieves all currently registered plugin types from the scheduler and dynamically renders them as draggable task nodes. This means that to extend a new data source or operation, developers only need to develop a compliant plugin and deploy it to the executor node; the entire process requires no modification, compilation, or restart of the core scheduling platform.

[0039] Furthermore, this application incorporates a resource-aware intelligent orchestration mechanism when constructing the DAG task flow. This mechanism allows users to call the resource measurement service (used for real-time monitoring of existing resources) API in real time to obtain the current cluster's real-time resource load (such as CPU / memory availability). Based on this resource information, the system performs real-time prediction and simulation of the user-orchestrated task flow. For example, highlighting can alert users to task nodes that may become bottlenecks due to insufficient resources, or provide resource consumption prompts when users connect to task dependencies, achieving a "what you see is what you get" intelligent orchestration.

[0040] After constructing the DAG task flow, it needs to be stored for later use in the scheduling platform during task scheduling. This application implements the storage of the DAG task flow using a structural class graph. The structural class graph of the DAG task flow includes at least one of the following: graph class, node class, edge information class, node data class, node port class, edge source information class, and edge target information class. See [link to relevant documentation]. Figure 3 As shown. Figure 3 Please refer to the table below for specific details on each type: Table 1. Description of Figures Figure 3 For detailed descriptions of various items, please refer to Table 2-8: Table 2 Node Data Class Description Table 3 Node Port Class Description Table 4 Node Class Description Table 5 Description of Edge-Source Information Table 6 Description of Side Target Information Classes Table 7. Description of Edge Information Classes Table 8. Description of Graphic Information The enumeration information in the aforementioned tables is shown in Table 9: Table 9 Enumeration Information See Figure 3As described in the above structural diagram, when the constructed DAG task flow is stored in the form of a structural diagram, the structural diagram is a multi-level graph structure used to store the graph information of the DAG task flow hierarchically. It should be noted that the first level of the structural diagram is the graph class, the second level is the node class and edge information class, and the third level is the node data class, node port class, edge source information class, and edge target information class. Specifically, when storing the DAG task flow, the node set and edge set of the DAG task flow are stored in the graph class. The information of each node in the node set is stored in the node class, and the node information includes the node ID, node type description, the X and Y coordinates displayed on the front end, the node data, and the node port. Detailed data of the node is stored in the node data class, and detailed data of the node port is stored in the node port class. The information of each edge in the edge set is stored in the edge information class, and the edge information includes the edge ID, edge type description, edge source, and edge target. The source details of an edge are stored in the edge source information class, and the target details of an edge are stored in the edge target information class.

[0041] Based on the above operations, the DAG task flow constructed by the front-end user can be stored in the form of a structural class diagram. When adding new task nodes in the future, it is only necessary to add the corresponding information of the new task type directly to the node class and edge information class in the structural class diagram. There is no need to modify the database, which can improve the scalability of the task scheduling system.

[0042] It should also be noted that the above structural diagram can also be converted into a JSON data structure for storage. See [link to documentation] for details. Figure 4 As shown. Since the JSON data structure is another form of class diagram representation, it is also easier to make changes such as adding or deleting tasks in the workflow.

[0043] After obtaining the DAG task flow, the next step is to retrieve each task node and its execution order based on the DAG task flow. Using a class graph structure for storage, each task node can be directly obtained from the node information in the graph, and the execution order of each task node can be obtained based on the edge information.

[0044] In one possible implementation, obtaining the execution order among task nodes includes: constructing dependencies between task nodes based on the DAG task flow; and determining the execution order of task nodes according to the dependencies. The execution order includes at least one of parallel execution and sequential execution.

[0045] In one possible implementation, the dependencies between task nodes are obtained based on the edge information, edge source information, and edge target information of the class graph. Then, based on these dependencies, the nodes with dependencies are executed in a sequential order, while the task nodes without dependencies can be executed in parallel order, thereby generating the execution order of the task flow.

[0046] Based on the generated task execution order, this application constructs an execution class graph for the task flow when executing the constructed DAG task flow, thereby realizing the execution of the task flow through the execution class graph. Specifically, when constructing the execution class graph, this application first loads the classes that the execution class graph depends on (i.e., the dependency library of the execution class graph, which contains the classes that the execution class graph depends on) through a custom core class loader (CoreClassLoader). It should be noted that when the core class loader loads the classes that the execution class graph depends on, it does not load the dependencies corresponding to each task node (i.e., the classes that each task node depends on). Instead, during actual execution, the dependencies corresponding to each task node are loaded independently through a custom plugin class loader (PluginClassLoader). Traditional task scheduling and execution typically uses a parent delegation mechanism to implement class loading; that is, after receiving a class loading request, the request is first delegated to the parent class loader for loading, and only when the parent class loader cannot load the class is the child class loader used for loading. See also... Figure 5 As shown, this application breaks with the traditional parent delegation mechanism, directly using subclass loaders to load the required classes, while the parent class loader only needs to load the classes it needs. Loading is only delegated to the parent class loader when a subclass loader is unable to load its own class. It should be noted that the custom core class loader and plugin class loader in this application both inherit from the abstract base class `ClassLoader`, and the plugin class loader uses the core class loader as its parent class loader.

[0047] In one possible implementation, breaking the parent delegation mechanism can be achieved using the following execution code: public class PluginClassLoader extends URLClassLoader { / / Load plugin dependencies first protected Class<?> loadClass(String name, boolean resolve) throwsClassNotFoundException { synchronized (getClassLoadingLock(name)) { / / 1. Check for packages that are prohibited from being loaded by the core system (security isolation) if (name.startsWith("com.platform.internal.")) { throw new ClassNotFoundException("Forbidden package:" + name); } / / 2. Prioritize loading classes from the plugin's own JAR file (innovation) Class<?> c = findLoadedClass(name); if (c == null) { try { c = findClass(name); / / Load from plugin JAR } catch (ClassNotFoundException e) { / / 3. If not found, delegate to the parent loader. c = super.loadClass(name, false); } } if (resolve) { resolveClass(c); } return c; } } } In one possible implementation, see [link to relevant documentation] Figure 6 The execution class diagram shown in this application is a multi-threaded execution diagram built based on the classes loaded by the core loader. This execution class diagram ensures that each task node is executed sequentially according to the execution order. The various descriptions in this execution class diagram are shown in Table 10. Table 10 Execution Class Diagram Description Based on Table 10 and Figure 6The steps of executing the class graph are explained below. Specifically, an execution instance of a DAG task flow (i.e., the DAG graph execution structure) is created, and the class methods in the DAG graph execution structure are called to obtain the task nodes and dependencies in the DAG graph. An executable execution graph is generated based on these dependencies. Each executable task node in the execution graph corresponds to a DAG worker. Each worker can call the execution logic corresponding to each task node (i.e., implemented through the `run` method) to execute. Each DAG worker is wrapped by a corresponding DAG worker wrapper class, which contains the data required for the DAG worker's execution. The DAG worker then executes the generated execution graph. In the execution graph, the tasks wrapped by each DAG worker are added to the thread pool in the execution order for execution. Upon completion, the execution result is fed back through a callback function. When the next task node is executed, the execution result of the previous task node that the current task node depends on is obtained through the data context, and execution is performed based on the result.

[0048] The parsing and scheduling process of the above task flow can be referenced in the following execution code: public class DagRunner { public void run(Graph graph) { / / Topological sorting to get the execution order List <node>executionOrder = topologicalSort(graph.nodes); / / Create a thread pool executor DagThreadPoolExecutor executor = new DagThreadPoolExecutor(); / / Build the task dependency tree Map<String, DagWorkerWrapper> nodeMap = new HashMap<>(); for (Node node : executionOrder) { DagWorker worker = new PluginWorker(node); / / Wrap the plugin execution logic DagWorkerWrapper wrapper = new DagWorkerWrapper(worker); / / Set dependencies for (Edge edge : graph.getIncomingEdges(node.id)) { DagWorkerWrapper dep = nodeMap.get(edge.source.cell); wrapper.addDependency(dep); } nodeMap.put(node.id, wrapper); executor.addToQueue(wrapper); } } } In one possible implementation, the specific logic executed by each task node can be converted into... Figure 7 The flowchart shown. See also... Figure 7 As shown, when executing each task node in the execution order, the process includes: obtaining the plugin corresponding to the current task node based on the execution order, and executing the execution logic contained in the plugin to obtain the execution result; determining whether to continue executing the execution logic of the plugin corresponding to the next task node based on the execution result; and continuing to execute the execution logic of the plugin corresponding to the next task node when the execution result is determined to be successful.

[0049] Specifically, see Figure 7 and Figure 8 As shown, the process first checks the prerequisites of the current task node. Specifically, it retrieves the execution results of the migration task nodes of the current task node according to the execution order (i.e., obtains the execution results from the data context). If the execution result is successful, the dependencies for the current task node need to be loaded. After loading, the dependencies for executing the execution logic of the current task node are obtained. The execution logic corresponding to the current task node is executed according to the loaded dependencies, and the execution result is obtained after execution. Then, based on the execution result, it is determined whether to trigger the execution logic of the next task node.

[0050] In one possible implementation, when determining whether to trigger the execution logic of the next task node based on the execution result, the process includes: if the execution result is successful, triggering the execution logic of the next task node; if the execution result is unsuccessful, attempting to execute again, and after three executions, if the execution result is still unsuccessful, marking the task node as failed and issuing a warning. It should be noted that regardless of whether the execution result is successful or unsuccessful, the visual status of the current task node on the front end will be updated to monitor the execution process of the task flow in real time.

[0051] It should be noted that after loading the dependencies corresponding to each task node, this application creates a security sandbox, and then executes the execution logic corresponding to the current task node within the created security sandbox. This is to ensure that when task nodes are running in parallel, each task node can execute independently within its own security sandbox.

[0052] The specific execution code for each task node can be seen as follows: public class PluginWorker extends DagWorker { public void run() { / / Retrieve the corresponding type of plugin from the plugin repository PluginInterface plugin = PluginRegistry.getPlugin(node.data.type); / / Execute in the security sandbox AccessController.doPrivileged( (PrivilegedAction <void>) () ->{ / / Construct the execution context (including input and output data) PluginContext context = new PluginContext(); context.setInputs(getInputDataFromPorts()); / / Execute the main logic of the plugin plugin.execute(context); / / Pass the output to downstream nodes setOutputDataToPorts(context.getOutputs()); return null; }, sandboxContext ); } } It should also be noted that the security sandboxes also have a dynamic minimum privilege set, used to allow users with different permissions to access different loaded resources. The dynamic minimum privilege set for each security sandbox can be dynamically set according to the actual user permissions, and there is no specific limitation. This can be represented by executable code as follows: / / Dynamically generate the minimum privilege set private PermissionCollection createPluginPermissions() { Permissions permissions = new Permissions(); / / Only grant necessary permissions permissions.add(new FilePermission(" / data / input / ", "read")); permissions.add(new SocketPermission("127.0.0.1:3306", "connect")); return permissions. } / / Apply sandbox during execution AccessController.doPrivileged( (PrivilegedAction <void>) () ->plugin.execute(context), new AccessControlContext( new ProtectionDomain[] { new ProtectionDomain new CodeSource(null, (Certificate[]) null), createPluginPermissions() / / Apply minimum permissions ) } ) ); It should also be noted that constraints are set when loading the execution logic of the plugins corresponding to each task node. These constraints are used to load the plugin's dependencies based on the constraints. These constraints include prioritizing the loading of dependencies by the plugin class loader when it can load its own dependencies, and simultaneously checking whether each class in the dependency is a class prohibited from loading by the core loader. If it is determined that the currently loaded class is a class prohibited from loading by the core loader, its loading is skipped. If the plugin class loader cannot load a class, the loading is delegated to the core loader.

[0053] The constraints, when executed, can be represented by the following executable code: public class PluginClassLoader extends URLClassLoader { protected Class<?> loadClass(String name, boolean resolve) { / / 1. Check if the core is blocking the loading of classes. if (isForbiddenClass(name)) throw new SecurityException(); / / 2. Prioritize loading from plugin JARs (breaking parent delegation) try { return findClass(name); } catch (ClassNotFoundException e) { / / 3. Delegate to the parent loader (core system) return super.loadClass(name, false); } } } Based on the above operations, all task nodes in the DAG task flow have completed execution, and the current task scheduling based on the constructed DAG task flow has ended.

[0054] It should also be noted that this application sets up a state-driven dynamic rendering mechanism. That is, this application connects the front-end designer directly or indirectly to the event bus via technologies such as WebSocket. When task state events flow through the bus, the front-end receives and updates the DAG view in real time. Node colors, shapes, progress bars, and log outputs will all change dynamically, providing an immersive monitoring experience.

[0055] Therefore, this application provides a DAG-based task flow execution method. The method includes obtaining the currently executed DAG task flow from pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structural class diagram; extracting each task node and the execution order between task nodes according to the DAG task flow; each task node represents a task type; searching for the corresponding plugin for each task node from a pre-built plugin library based on the task type; executing each plugin sequentially according to the execution order; and concluding the execution of the DAG task flow after each plugin has finished executing. This application, by storing the constructed DAG task flow as a structural class diagram, allows for direct classification and storage of new task type information in the structural class diagram when new data sources or task types need to be added, effectively improving the scalability of the DAG task scheduling system. Furthermore, this application packages the execution logic corresponding to each task type into independent plugins, allowing each task to run independently during execution. This allows the application to directly package new data sources or task types into corresponding new plugins and store them in a plugin library when expanding to new data sources or task types. After registering the task types corresponding to the new plugins in the plugin library, they can be freely added to the existing DAG task flow for execution. This process only requires registering the task types corresponding to the new plugins, eliminating the need for complex operations and improving the scalability of the DAG task scheduling system. Furthermore, through its plugin-based design, this application loads the dependencies required for core platform execution and the dependencies of each task node independently during dependency loading, breaking away from the traditional parent delegation mechanism and effectively improving the security performance of the task scheduling platform.

[0056] <Device Embodiment> Figure 9 A schematic block diagram of a DAG-based task flow execution apparatus according to an embodiment of this application is shown. Figure 9 As shown, the device 100 is used for, and the device 100 includes: a DAG task flow acquisition module 110, a DAG task flow data extraction module 120, a plugin search module 130, and a plugin execution module 140. The DAG task flow acquisition module 110 is used to acquire the DAG task flow to be executed from pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structural class diagram; the DAG task flow data extraction module 120 is used to extract each task node and the execution order between each task node according to the DAG task flow; each task node represents a task type; the plugin search module 130 is used to search for the plugin corresponding to each task node from a pre-built plugin library based on the task type; the plugin execution module 140 is used to execute each plugin sequentially according to the execution order, and the execution of the DAG task flow ends when each plugin is executed sequentially.

[0057] <Equipment Example> Figure 10 A schematic block diagram of a DAG-based task flow execution device according to an embodiment of this application is shown. Figure 10 As shown, the DAG-based task flow execution device 200 includes a processor 210 and a memory 220 for storing executable instructions of the processor 210. The processor 210 is configured to implement any of the aforementioned DAG-based task flow execution methods when executing executable instructions.

[0058] It should be noted here that the number of processors 210 can be one or more. Furthermore, the DAG-based task flow execution device 200 in this embodiment may also include an input device 230 and an output device 240. The processors 210, memory 220, input device 230, and output device 240 can be connected via a bus or other means, which are not specifically limited here.

[0059] The memory 220, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the DAG-based task flow execution method in this application embodiment. The processor 210 executes various functional applications and data processing of the DAG-based task flow execution device 200 by running the software program or module stored in the memory 220.

[0060] Input device 230 can be used to receive input digital numbers or signals. These signals may be key signals related to user settings and function control of the device / terminal / server. Output device 240 may include a display device such as a screen.

[0061] <Storage Medium Examples> According to a fourth aspect of this application, a non-volatile computer-readable storage medium is also provided, on which computer program instructions are stored, which, when executed by processor 210, implement any of the aforementioned DAG-based task flow execution methods.

[0062] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.< / void> < / void> < / node>

Claims

1. A task flow execution method based on DAG, characterized in that, include: The DAG task flow to be executed is obtained from the pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structure class diagram; Based on the DAG task flow, each task node and the execution order between each task node are extracted; each task node is used to represent a task type. Based on the task type, the plugin corresponding to each task node is searched from a pre-built plugin library; Each plugin is executed sequentially according to the execution order, and the DAG task flow ends when each plugin finishes execution.

2. The task flow execution method based on DAG according to claim 1, characterized in that, When obtaining the execution order among the task nodes, the following is included: The dependency relationships between the task nodes are constructed based on the DAG task flow; The execution order of each task node is determined based on the dependencies.

3. The task flow execution method based on DAG according to claim 2, characterized in that, The execution order includes at least one of parallel execution and sequential execution.

4. The task flow execution method based on DAG according to claim 1, characterized in that, The plugin contains the execution logic for the corresponding task type.

5. A task flow execution method based on DAG according to any one of claims 1 to 4, characterized in that, When executing each of the plugins sequentially based on the execution order, the following is included: Based on the execution order, obtain the plugin corresponding to the current task node, and execute the plugin to obtain the execution result; Based on the execution result, determine whether to continue executing the next task node. If the execution result is determined to be successful, the plugin corresponding to the next task node will continue to be executed.

6. The task flow execution method based on DAG according to claim 1, characterized in that, When the DAG task flow is stored in the DAG task data in the form of a structural class graph, the structural class graph includes at least one of the following: graph class, node class, edge information class, node data class, node port class, edge source information class, and edge target information class.

7. The task flow execution method based on DAG according to claim 1, characterized in that, The DAG task flow is stored in the form of a structural class diagram before the DAG task data, and also includes the operation of constructing the DAG task flow.

8. A task flow execution device based on DAG, characterized in that, include: The DAG task flow acquisition module is used to acquire the DAG task flow to be executed from the pre-stored DAG task data, wherein the DAG task flow is stored in the DAG task data in the form of a structure class diagram; The DAG task flow data extraction module is used to extract each task node and the execution order between each task node according to the DAG task flow; each task node is used to represent a task type. The plugin search module is used to search for the plugins corresponding to each task node from a pre-built plugin library based on the task type. The plugin execution module is used to execute each plugin sequentially according to the execution order. Once each plugin has finished executing, the DAG task flow execution ends.

9. A task flow execution device based on DAG, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 7 when executing the executable instructions.

10. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images