Task scheduling and executing method, device and system, electronic equipment and storage medium

Abstract

The invention discloses a task scheduling and executing method, device and system, electronic equipment and a storage medium. The task scheduling and executing method comprises the steps that a target data flow diagram is obtained, the target data flow diagram comprises a plurality of nodes, and the nodes comprise a first type of nodes; analyzing the target data flow diagram, and generating task instances corresponding to the first type of nodes; generating a dependency relationship between task instances according to the dependency relationship between the first type of nodes in the target data flow diagram, and obtaining a target task flow diagram based on the dependency relationship between the task instances; and performing task scheduling and execution based on the target task flow graph.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, and particularly relates to a task scheduling and execution method, device, system, electronic equipment and storage medium. BACKGROUND

[0002] In a complex data processing process, there are data dependency relationships and execution order relationships between tasks, and therefore effective management and scheduling are needed.

[0003] In the related art, when performing task processing, a corresponding subtask is usually generated based on each node in a data flow graph, and each subtask is executed in turn. When each subtask is executed, the execution result corresponding to each subtask needs to be stored in a data storage space, so as to perform data query when the next subtask is executed. As can be seen, storing the execution result of each node corresponding subtask occupies too much storage space, and the intelligent degree of task processing is also low. SUMMARY

[0004] In view of the above problems, the present application provides a task scheduling and execution method, device, system, electronic equipment and storage medium.

[0005] In a first aspect, the present application provides a task scheduling and execution method, which comprises: acquiring a target data flow graph, wherein the target data flow graph comprises a plurality of nodes, and the plurality of nodes comprise a first type of node; analyzing the target data flow graph to generate a task instance corresponding to the first type of node; generating a dependency relationship between task instances according to a dependency relationship between the first type of nodes in the target data flow graph, and obtaining a target task flow graph based on the dependency relationship between the task instances; and performing task scheduling and execution based on the target task flow graph.

[0006] In the technical scheme of the present application, first, a target data flow graph is acquired, wherein the target data flow graph comprises a plurality of nodes, and the plurality of nodes comprise a first type of node. Next, the target data flow graph is analyzed to generate a task instance corresponding to the first type of node. Then, a dependency relationship between task instances is generated according to a dependency relationship between the first type of nodes in the target data flow graph, and a target task flow graph is obtained based on the dependency relationship between the task instances. Finally, task scheduling and execution are performed based on the target task flow graph.

[0007] The target data flow graph indicates the nodes and their dependencies, accurately defining the processing flow of task instances. By directly parsing the target data flow graph, the task instances corresponding to the first type of nodes and their dependencies can be obtained. Task scheduling and execution based on the target task flow graph are simple and fast, requiring minimal manual intervention. The automation and intelligence of data processing are high, improving both efficiency and accuracy. Furthermore, based on node type, only the first type of nodes in the target data flow graph need to be instantiated. During task scheduling and execution, only the execution results of the task instances corresponding to the first type of nodes need to be stored, eliminating the need to instantiate and store the execution results of all nodes in the target data flow graph, thus reducing data storage space requirements.

[0008] In some embodiments, the plurality of nodes further includes a second type of node, and the method further includes: in the step of parsing the target data flow graph and generating a task instance corresponding to the first type of node, generating a data processing logic statement corresponding to the second type of node; determining the first type of node at the next level of the second type of node according to the dependency relationship between the second type of node and the first type of node in the target data flow graph, and adding the data processing logic statement to the task instance corresponding to the first type of node at the next level, so that when the corresponding task instance is executed during task scheduling and execution based on the target task flow graph, the data processing logic statement is executed first to obtain the input data of the task instance.

[0009] In the technical solution of this application embodiment, the target data flow graph is parsed to generate data processing logic statements corresponding to the second type of nodes, without the need to generate separate task instances. Based on the dependency relationship between the second type of nodes and the first type of nodes in the target data flow graph, the next-level first type of nodes of the second type of nodes are determined, and the data processing logic statements are added to the task instances corresponding to these next-level first type of nodes. This ensures that during task scheduling and execution based on the target task flow graph, when the corresponding task instance is encountered, the data processing logic statements are executed first to obtain the input data for the task instance. Therefore, merging the data processing logic statements into the task instances corresponding to the next-level first type of nodes avoids generating too many task instances, preventing excessive dependencies and cumbersome scheduling processes. Furthermore, the execution results generated by the data processing logic statements do not need to be stored; the execution results are released directly after the data processing logic statements are executed, saving storage resources.

[0010] In some embodiments, generating the dependency relationship between task instances based on the dependency relationship between the first type of nodes in the target data flow graph includes: for two first type nodes with a dependency relationship in the target data flow graph, if there is no second type node between the two first type nodes, determining the dependency relationship between the two first type nodes as the dependency relationship between the two task instances corresponding to the two first type nodes; and / or if there is a second type node between the two first type nodes, obtaining the direct dependency relationship between the two first type nodes based on the dependency relationship between the second type node and the two first type nodes respectively, and determining the direct dependency relationship as the dependency relationship between the two task instances corresponding to the two first nodes.

[0011] In the technical solution of this application embodiment, determining the dependencies between task instances helps to optimize the task scheduling and execution process.

[0012] In some embodiments, the task scheduling and execution based on the target task flow graph includes: scheduling task instances from the target task flow graph according to the dependencies between task instances; for a scheduled task instance, obtaining the input data of the task instance based on the output data of the previous level task instance, and executing the task instance.

[0013] In the technical solution of this application embodiment, input data is a key component of task instance execution, ensuring that the task instance can run as expected and produce correct output results.

[0014] In some embodiments, obtaining the input data of a scheduled task instance based on the output data of its parent task instance includes: for a scheduled task instance, if the task instance includes data processing logic statements, processing the output data of the parent task instance based on the data processing logic statements to obtain a processing result; and using the processing result and / or the output data of the parent task instance as the input data of the task instance.

[0015] In the technical solution of this application embodiment, input data is a key component of task instance execution, ensuring that the task instance can run as expected and produce correct output results.

[0016] In some embodiments, the task scheduling and execution based on the target task flow graph includes: querying task instances in the target task flow graph that are in a successfully created state; responding to task triggering, sequentially scheduling and executing each task instance in the successfully created state according to the dependency relationship between task instances in the target task flow graph; wherein, for each scheduled task instance, if the parent task instance to which the task instance depends has been executed, the task instance is sent to the corresponding task executor for execution; if the parent task instance to which the task instance depends has not been executed, the task instance is placed in the task pool to wait.

[0017] In some embodiments, the method further includes: for each scheduled task instance, when executing the task instance, if the task instance is successfully executed, updating the state of the task instance to a success state and continuing to schedule the execution of the next-level task instances that depend on the task instance; if the task instance fails to execute, updating the state of the task instance to a failure state and updating the state of all subsequent task instances that depend on the task instance to a failure state.

[0018] In the technical solution of this application embodiment, when the status of a task instance is updated to a successful state, the next-level task instance is scheduled to ensure the continuity and consistency of the task instance chain. When the status of a task instance is updated to a failed state, the failed task instance will cause all subsequent task instances that depend on it to be updated to a failed state, which helps to promptly detect and resolve problems and prevent failed task instances from continuing to affect the system.

[0019] In some embodiments, the task instance includes at least data input, output associated query statements, output result table information, field information, and running parameters. Executing a task instance includes: for each scheduled task instance, when executing the task instance, the task executor executes according to the running parameters of the task instance.

[0020] In the technical solution of this application embodiment, by determining the data input, output associated query statements, output result table information, field information, and running parameters included in the task instance, it is ensured that the execution of the task instance has a clear goal and process. When executing the task instance, the task executor executes according to the running parameters of the task instance, thereby achieving personalized processing and optimization.

[0021] In some embodiments, for each scheduled task instance, when executing the task instance, the process includes: sending the task instance to the corresponding task executor for execution if there are sufficient idle resources in the resource queue of the corresponding task executor; and / or placing the task instance in the task pool to wait if there are higher priority tasks waiting for resources.

[0022] In the technical solution of this application embodiment, by determining whether there are sufficient idle resources in the resource queue of the corresponding task executor, resources can be effectively managed and allocated, enabling task instances to be executed promptly when resources are sufficient and to wait when resources are insufficient. By determining whether there are higher-priority task instances waiting for resources, resource allocation and scheduling strategies can be effectively optimized, ensuring that high-priority task instances can be executed first, thereby improving the efficiency and responsiveness of task instance execution.

[0023] In some embodiments, for each scheduled task instance, when executing the task instance, the process includes: sending the task instance to the task executor corresponding to the task type for execution based on the task type of the task instance, wherein the task type includes any one of data calculation type and data cleaning type, wherein if the task instance includes data processing logic statements, during the execution of the task instance, the data processing logic statements are sent to the task executor corresponding to the data processing logic statements for execution.

[0024] In the technical solution of this application embodiment, a suitable task executor can be intelligently selected for execution based on the task type of the task instance. This not only improves the efficiency of task instance processing but also ensures that each type of task instance is executed by a dedicated task executor, thereby fully utilizing the specific capabilities and advantages of the task executor.

[0025] In some embodiments, parsing the target data flow graph includes: for any one of the plurality of nodes in the target data flow graph, before parsing the node to obtain the task instance corresponding to the node or converting the node into a data processing logic statement, traversing all the parent nodes of the node in a breadth-first traversal manner, and after parsing all the parent nodes to obtain the output data of all the parent nodes, using the output data as the input data of the node and parsing the node.

[0026] In the technical solution of this application embodiment, for any node among multiple nodes in the target data flow graph, before parsing the node to obtain the corresponding task instance or converting the node into a data processing logic statement, all parent nodes of the node are traversed in a breadth-first traversal manner. After parsing all parent nodes to obtain their output data, the output data is used as the input data for the node and the node is parsed. This ensures that the output data of all parent nodes to which the node depends are fully collected, thereby avoiding the omission of parent data and ensuring that the node is correctly parsed.

[0027] In some embodiments, the method further includes: obtaining a workflow, wherein the workflow includes target node information, edge information between target nodes, and configuration information of target nodes; and constructing the target data flow graph based on the workflow.

[0028] In the technical solution of this application embodiment, a general target data flow graph is constructed based on the workflow, which is flexible and convenient.

[0029] In some embodiments, the workflow is generated by: outputting candidate nodes to an orchestration interface; receiving orchestration operations for the candidate nodes through the orchestration interface to obtain information about the target node, edge information between nodes, and configuration information of the target node; and obtaining the workflow based on the information about the target node, the edge information, and the configuration information of the target node.

[0030] In the technical solution of this application embodiment, candidate nodes are output to the orchestration interface. The orchestration interface receives orchestration operations for the candidate nodes, obtaining information about the target node, edge information between nodes, and configuration information of the target node. This process is simple to operate and reduces development difficulty. Based on the target node information, edge information, and configuration information, a workflow is obtained, and a universal target data flow graph is constructed. This process is flexible and convenient.

[0031] In some embodiments, receiving orchestration operations for the candidate nodes through the orchestration interface to obtain information about the target node, edge information between nodes, and configuration information of the target node includes: receiving a selection operation through the orchestration interface and selecting a target node from the candidate nodes based on the selection operation to obtain information about the target node, wherein the target node includes at least one of a data source node, a data processing node, and an operator node; outputting the target node to the orchestration interface; receiving a connection operation performed on the target node through the orchestration interface and generating edges between the target nodes based on the connection operation to obtain the edge information; and receiving a configuration operation for the target node through the orchestration interface and generating configuration information of the target node based on the configuration operation.

[0032] In the technical solution of this application embodiment, the orchestration interface provides a visual interface, on which users can quickly select target nodes and perform connection and configuration operations to obtain target node information, edge information, and target node configuration information. This process is simple to operate, reduces development difficulty, and thus improves the efficiency of workflow construction. Users can manage various types of nodes in an intuitive way, realizing batch scheduling of nodes and full lifecycle management of tasks.

[0033] On the other hand, this application provides a task scheduling and execution apparatus, characterized in that the apparatus includes: an acquisition and parsing module, used to acquire and parse a target data flow graph to generate multiple task instances corresponding one-to-one with multiple nodes in the target data flow graph, and to generate dependency relationships between task instances based on the dependency relationships between nodes in the target data flow graph; and a scheduling and execution module, used to query all task instances in the successfully created state among the multiple task instances generated based on the target data flow graph, and in response to task triggering, to schedule and execute each task instance in the successfully created state sequentially according to the arrow direction of the target data flow graph, starting from the data source node, wherein, for each scheduled task instance, it is determined whether all the parent task instances that the scheduled task instance depends on have been executed; if so, the scheduled task instance is sent to the corresponding task executor for execution; otherwise, the scheduled task instance is placed in the task pool to wait.

[0034] On the other hand, this application provides a task scheduling and execution system, which includes a task scheduling system and a task executor. The task scheduling system and the task executor are communicatively connected, and the task scheduling and execution system is capable of performing the steps of the method according to any of the above claims.

[0035] On the other hand, this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the above embodiments.

[0036] On the other hand, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0037] On the other hand, this application provides a computer program product that includes instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the steps of the method described in any of the above embodiments.

[0038] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1a A flowchart illustrating the task scheduling and execution method provided for embodiments of this application;

[0041] Figure 1b A schematic diagram of the system application architecture provided for the embodiments of this application;

[0042] Figure 2a A schematic diagram illustrating the scheduling and execution process of the second type of node provided in the embodiments of this application;

[0043] Figure 2b A schematic diagram illustrating the input data for obtaining the task instance corresponding to Model 2, provided for an embodiment of this application;

[0044] Figure 3 A flowchart illustrating task scheduling and execution provided for embodiments of this application;

[0045] Figure 4 A flowchart illustrating the process of obtaining input data for a task instance, provided for an embodiment of this application;

[0046] Figure 5 A schematic diagram illustrating the execution of a data JOIN node as provided in the embodiments of this application;

[0047] Figure 6 This illustration shows a flowchart of constructing the target data flow graph according to an embodiment of this application;

[0048] Figure 7 A schematic diagram of the workflow obtained according to an embodiment of this application is shown;

[0049] Figure 8a This paper illustrates a flowchart of the process for generating configuration information for a target node according to an embodiment of this application.

[0050] Figure 8b A schematic diagram of the arrangement interface according to an embodiment of this application is shown;

[0051] Figure 8c A schematic diagram of the node configuration in an embodiment of this application is shown;

[0052] Figure 9 A schematic diagram of the workflow obtained according to an embodiment of this application is shown;

[0053] Figure 10This document illustrates a flowchart of the task instance creation and scheduling process according to an embodiment of this application.

[0054] Figure 11 A schematic diagram of a task scheduling and execution device provided for an embodiment of this application. Detailed Implementation

[0055] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0057] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0060] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0061] In the description of the embodiments of this application, the technical terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0062] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0063] By acquiring the input and output data of the Directed Acyclic Graph (DAG) and nodes, the execution time is predicted. Based on the prediction results, the first and second schedules are generated to complete the task scheduling of data unloading and prefetching, thereby optimizing and improving the efficiency of machine learning.

[0064] When there are data dependencies and sequential calling relationships between multiple data processing tasks and operator tasks, such as when the input data of one operator comes from the output results of one or more other operators, it is necessary to configure, orchestrate and schedule multiple operators to perform calculations, process the output data of the dependent operator into the input data of other operators, and execute the dependent operator after the dependent operator has finished executing.

[0065] In view of this, the embodiments of this application provide a task scheduling and execution method.

[0066] Figure 1a A flowchart illustrating the task scheduling and execution method provided in the embodiments of this application.

[0067] like Figure 1a As shown, the task scheduling and execution method 100 provided in this application includes, for example, steps S110-S140.

[0068] Step S110: Obtain the target data flow graph.

[0069] Step S120: Parse the target data flow graph and generate task instances corresponding to the first type of nodes.

[0070] The target data flow graph includes multiple nodes, including the first type of nodes.

[0071] For example, a target data flow graph is obtained based on the workflow, and each node in the target data flow graph is instantiated according to the target data flow graph to perform subsequent operations. The nodes of the target data flow graph are traversed (e.g., a width traversal), and the node type and edge information are determined by parsing the target data flow graph. Whether instantiation is necessary depends on the node type. If the node is a first-type node, it needs to be instantiated. A specific instance is created based on the definition and configuration of the first-type node, generating the corresponding task instance. The first-type node can be a data processing node (e.g., data cleaning, data synchronization) or an operator node.

[0072] After instantiating all first-type nodes in the target data flow graph, multiple task instances and their dependencies are generated, each corresponding one-to-one with a first-type node in the target data flow graph. The instantiated task instances and their dependencies are then stored in a database.

[0073] For example, Xxl-job is a lightweight distributed task scheduling platform that initiates workflow execution by triggering a scheduling system through scheduled tasks. Xxl-job can be replaced by other scheduled task applications or code libraries (such as Quartz). Under the trigger of Xxl-job, the execution logic of the scheduled task instances is triggered and managed by external scheduled tasks. See also... Figure 1b The task scheduling system is based on a front-end page and stores the workflow in an open-source relational database management system (MySQL). The task scheduling system queries the workflow from the MySQL database and processes it to obtain the target data flow graph.

[0074] The task instances corresponding to the first type of nodes obtained by instantiating the target data flow graph can be Spark tasks, SQL tasks, or AI tasks. Multiple task instances and their dependencies can be stored in a MySQL database.

[0075] It should be noted that a workflow can be instantiated and run multiple times, for example, multiple times a day. The generated task instances need to be persistently stored to ensure that each instantiation runs the task based on the latest definitions and parameters.

[0076] Step S130: Generate the dependency relationship between task instances based on the dependency relationship between the first type of nodes in the target data flow graph, and obtain the target task flow graph based on the dependency relationship between task instances.

[0077] Step S140: Perform task scheduling and execution based on the target task flow graph.

[0078] For example, while generating task instances, the execution order of the task instances is determined based on the dependencies between the first type of nodes, and the dependencies between task instances are generated. A target task flow graph is constructed using the task instances and their dependencies. The dependencies between task instances form the execution model, guiding task scheduling and execution. Based on the target task flow graph, task scheduling and execution are performed considering factors such as priority and resource allocation.

[0079] In the technical solution of this application embodiment, firstly, a target data flow graph is obtained, which includes multiple nodes, including first-type nodes. Next, the target data flow graph is parsed to generate task instances corresponding to the first-type nodes. Then, dependencies between task instances are generated based on the dependencies between the first-type nodes in the target data flow graph, and a target task flow graph is obtained based on these dependencies. Finally, task scheduling and execution are performed based on the target task flow graph.

[0080] The target data flow graph indicates the nodes and their dependencies, accurately defining the processing flow of task instances. By directly parsing the target data flow graph, the task instances corresponding to the first type of nodes and their dependencies can be obtained. Task scheduling and execution based on the target task flow graph are simple and fast, requiring minimal manual intervention. The automation and intelligence of data processing are high, improving both efficiency and accuracy. Furthermore, based on node type, only the first type of nodes in the target data flow graph need to be instantiated. During task scheduling and execution, only the execution results of the task instances corresponding to the first type of nodes need to be stored, eliminating the need to instantiate and store the execution results of all nodes in the target data flow graph, thus reducing data storage space requirements.

[0081] Figure 2a This is a schematic diagram illustrating the scheduling and execution process of the second type of node provided in the embodiments of this application.

[0082] like Figure 2a As shown, the scheduling and execution method 200 for the second type of node provided in this application includes steps S210-S220.

[0083] In step S210, during the step of parsing the target data flow graph and generating task instances corresponding to the first type of nodes, data processing logic statements corresponding to the second type of nodes are also generated.

[0084] Step S220: Based on the dependency relationship between the second type of nodes and the first type of nodes in the target data flow graph, determine the first type of nodes at the next level of the second type of nodes, and add the data processing logic statement to the task instance corresponding to the first type of node at the next level, so that when the corresponding task instance is executed during task scheduling and execution based on the target task flow graph, the data processing logic statement is executed first to obtain the input data of the task instance.

[0085] For example, a target data flow graph is obtained based on the workflow, and each node in the target data flow graph is instantiated according to the target data flow graph to execute subsequent operations. The nodes of the target data flow graph are traversed (e.g., width traversal), and the node type and edge information are determined by parsing the target data flow graph. Instantiation is determined based on the node type. If the node is a second-type node, data processing logic statements (e.g., filtering SQL statements, UNION statements, JOIN statements) corresponding to the second-type node are generated. While generating the data processing logic statements, the dependency relationships between the second-type and first-type nodes are used to determine which task instances of the first-type nodes need to depend on the processing results of the data processing logic statements corresponding to the second-type nodes. The data processing logic statements corresponding to the second-type nodes are added to the task instances corresponding to the first-type nodes at the next level of the second-type nodes. During task scheduling and execution based on the target task flow graph, when the corresponding task instance is executed, the data processing logic statements are executed first to obtain the input data of the task instance. The second-type node can be a data source node, a data filtering node, a data join node, or a data combination UNION node.

[0086] For example, please refer to Figure 2b Add the data processing logic statements corresponding to the data filtering node and the data join node to the task instance corresponding to Model 2. When the task instance corresponding to Model 2 is executed, first execute the data processing logic statements corresponding to the data filtering node and the data join node to implement data filtering and data join. At this time, there is no need to store the results of data filtering and data join in the database. After the execution is completed, they are released directly, thus obtaining the input data of the task instance corresponding to Model 2.

[0087] In the technical solution of this application embodiment, the target data flow graph is parsed to generate data processing logic statements corresponding to the second type of nodes, without the need to generate separate task instances. Based on the dependency relationship between the second type of nodes and the first type of nodes in the target data flow graph, the next-level first type of nodes of the second type of nodes are determined, and the data processing logic statements are added to the task instances corresponding to these next-level first type of nodes. This ensures that during task scheduling and execution based on the target task flow graph, when the corresponding task instance is encountered, the data processing logic statements are executed first to obtain the input data for the task instance. Therefore, merging the data processing logic statements into the task instances corresponding to the next-level first type of nodes avoids generating too many task instances, preventing excessive dependencies and cumbersome scheduling processes. Furthermore, the execution results generated by the data processing logic statements do not need to be stored, saving storage resources.

[0088] In another example, generating dependencies between task instances based on dependencies between first-type nodes in the target data flow graph may include: for two first-type nodes with dependencies in the target data flow graph; if there are no second-type nodes between the two first-type nodes, the dependency between the two first-type nodes is determined as the dependency between the two task instances corresponding to the two first-type nodes; and / or if there are second-type nodes between the two first-type nodes, based on the dependencies between the second-type nodes and the two first-type nodes respectively, the direct dependency between the two first-type nodes is obtained, and the direct dependency is determined as the dependency between the two task instances corresponding to the two first-type nodes.

[0089] For example, since the second type of node does not generate task instances, the dependency relationship between the first type of node and the second type of node that depends on the second type of node is not stored during the instantiation of the first type of node. Therefore, in order for the dependency relationship between task instances to be correctly reflected in the target task flow graph, for two first type of nodes, if there is no second type of node between the two first type of nodes, but a direct dependency relationship exists, then the dependency relationship between the two first type of nodes is directly determined as the dependency relationship between the two task instances corresponding to the two first type of nodes.

[0090] If there is an indirect dependency relationship between the two first-type nodes and a second-type node, then based on the dependencies between the second-type node and each of the two first-type nodes, the direct dependency relationship between the two first-type nodes can be derived. This direct dependency relationship is then defined as the dependency relationship between the two task instances corresponding to the two first-type nodes.

[0091] In the technical solution of this application embodiment, determining the dependencies between task instances helps to optimize the task scheduling and execution process.

[0092] Figure 3 This is a flowchart illustrating the task scheduling and execution process provided for an embodiment of this application.

[0093] like Figure 3 As shown, the task scheduling and execution method 300 provided in this application includes steps S310-S320.

[0094] Step S310: Schedule task instances from the target task flow graph based on the dependencies between task instances.

[0095] Step S320: For the scheduled task instance, based on the output data of the previous level task instance, obtain the input data of the task instance and execute the task instance.

[0096] For example, the target task flow graph is queried from the data flow graph (DAG) scheduling system. In response to task triggering, the task instance corresponding to the data source node is used as the starting point. Then, according to the dependencies between task instances, each task instance is scheduled and executed sequentially from the target task flow graph. For each scheduled task instance, its input data is obtained based on the output data of its parent task instance, such as output result table information and field information. The task instance then executes based on the input data. Finally, the task instance pushes its own output data, such as output result table information and field information, down to the next-level task instances that depend on it, so that the next-level task instances can obtain the corresponding input data.

[0097] In the technical solution of this application embodiment, input data is a key component of task instance execution, ensuring that the task instance can run as expected and produce correct output results.

[0098] Figure 4 This is a schematic flowchart illustrating the process of obtaining input data for a task instance, provided as an embodiment of this application.

[0099] like Figure 4 As shown, the method 400 for obtaining input data of a task instance provided in this application includes steps S410-S420.

[0100] Step S410: For a scheduled task instance, if the task instance includes data processing logic statements, process the output data of the previous level task instance based on the data processing logic statements to obtain the processing result.

[0101] Step S420: Use the processing result and / or the output data of the previous level task instance as the input data of this task instance.

[0102] For example, for a scheduled task instance, if the task instance includes data processing logic statements, the output data of the parent task instance is queried as the basis for the data processing of this task instance. The output data of the parent task instance is processed based on the data processing logic statements to obtain the processing result. If the task instance does not have a dependent parent task instance, the processing result is used as the input data of the task instance. If the task instance has a dependent parent task instance, the processing result and / or the output data of the parent task instance are used as the input data of the task instance.

[0103] In the technical solution of this application embodiment, input data is a key component of task instance execution, ensuring that the task instance can run as expected and produce correct output results.

[0104] In another example, task scheduling and execution based on the target task flow graph may include: querying task instances in the target task flow graph that are in the successfully created state, and, in response to task triggering, scheduling and executing each task instance in the successfully created state in sequence according to the dependency relationship between task instances in the target task flow graph.

[0105] Specifically, for each scheduled task instance, if the parent task instance it depends on has been completed, the task instance is sent to the corresponding task executor for execution; if the parent task instance it depends on has not been completed, the task instance is placed in the task pool to wait.

[0106] For example, from the Data Flow Graph (DAG) scheduling system, query the task instances in the target task flow graph that are in the successfully created state. For instance, in response to a task trigger, starting with the task instance corresponding to the data source node, schedule and execute each successfully created task instance sequentially according to the dependencies between task instances in the target task flow graph. After each task instance is executed, update the task status and store the execution results in the database. For each scheduled task instance, before execution, it is necessary to check and determine whether all its dependent parent task instances have been executed. This can be done periodically using a polling mechanism to check the status of dependent parent task instances. If all dependent parent task instances have been executed, the task instance is sent to the corresponding task executor for execution. If not all dependent parent task instances have been executed, the task instance is placed in a task pool to wait until all parent task instances have been executed before being sent to the corresponding task executor for execution. For example, during each polling, if it is determined that at least one of the parent task instances that the task instance depends on has not been completed, the task instance will remain in a waiting state until it is determined that all the parent task instances that the task instance depends on have been completed, at which point the task instance will be sent to the corresponding task executor for execution.

[0107] For example, please see Figure 1b Data warehouse tools (Hive database) ) It can store the execution results of task instances. Depending on the data volume, data characteristics, and specific application scenarios, storage systems such as MySQL, ClickHouse, HBase, or Minio can be chosen to replace the Hive database.

[0108] In another example, for each scheduled task instance, when executing the task instance, if the task instance is executed successfully, the status of the task instance is updated to a success status, and the next-level task instances that depend on the task instance are scheduled for execution; if the task instance fails to execute, the status of the task instance is updated to a failure status, and the status of all subsequent task instances that depend on the task instance is updated to a failure status.

[0109] For example, for each scheduled task instance, when executing the task instance, the task executor returns the execution result of the task, which can be success or failure. If the task instance is executed successfully, its status is updated to a success status, and then, based on the dependencies between task instances in the target task flow graph, the next-level task instances that depend on this task instance are scheduled for execution. If the task instance fails to execute, its status is updated to a failure status, and then, based on the dependencies between task instances in the target task flow graph, all subsequent task instances that depend on this task instance are found, and the status of all subsequent task instances is updated to a failure status.

[0110] In the technical solution of this application embodiment, when the status of a task instance is updated to a successful state, the next-level task instance is scheduled to ensure the continuity and consistency of the task instance chain. When the status of a task instance is updated to a failed state, the failed task instance will cause all subsequent task instances that depend on it to be updated to a failed state, which helps to promptly detect and resolve problems and prevent failed task instances from continuing to affect the system.

[0111] In another example, a task instance includes at least the data input, the associated query statement, the output result table information, the field information, and the running parameters.

[0112] For example, the data table and field information corresponding to each task instance are stored in a database. The output result table information and field information of the parent task instance that the currently scheduled task instance depends on are determined. The output result table information and field information are combined to generate a query statement for the currently scheduled task instance. Using the query statement, input data for the currently scheduled task instance is generated based on the output result table information and field information of the parent task instance. The currently scheduled task instance executes based on the input data and runtime parameters. Simultaneously, the currently scheduled task instance pushes its own output result table information and field information down to the next-level task instances that depend on it, so that the next-level task instances can generate related query statements. The currently scheduled task instance includes the query statement associated with data input and data output, output result table information, field information, and runtime parameters. These runtime parameters are a key component of the execution process, ensuring that the task instance can run as expected and produce correct output results.

[0113] In the technical solution of this application embodiment, by determining the data input, output associated query statements, output result table information, field information and running parameters included in the task instance, it is ensured that the execution of the task instance has a clear goal and process.

[0114] In another example, for each scheduled task instance, when executing that task instance, the task executor executes according to the running parameters of that task instance.

[0115] For example, for each scheduled task instance, when executing the task instance, the task executor parses the task instance and executes the corresponding task according to the obtained running parameters.

[0116] In the technical solution of this application embodiment, when executing the task instance, the task executor executes according to the running parameters of the task instance, thereby realizing personalized processing and optimization.

[0117] In another example, for each scheduled task instance, when executing the task instance, it may include: if there are sufficient idle resources in the resource queue of the corresponding task executor, sending the task instance to the corresponding task executor for execution.

[0118] For example, for each scheduled task instance, when executing the task instance, it is necessary to determine whether there are sufficient idle resources in the resource queue of the task executor corresponding to the task type of the task instance. If the task executor has sufficient resources, the task instance is sent to the corresponding task executor for execution. If the task executor does not have sufficient resources, the task instance will be directly returned to the task pool to wait for subsequent scheduling.

[0119] In the technical solution of this application embodiment, by determining whether there are sufficient idle resources in the resource queue of the corresponding task executor, resources can be effectively managed and allocated, so that task instances can be executed in a timely manner when resources are sufficient, and wait when resources are insufficient.

[0120] In another example, for each scheduled task instance, when executing that task instance, it may include: if there are higher priority tasks waiting for resources, then put the task instance into the task pool to wait.

[0121] For example, for each scheduled task instance, when executing that task instance, it is necessary to determine whether there is a higher-priority task instance waiting for resources. If a higher-priority task instance exists, it will be scheduled first and assigned to the task executor corresponding to the task type for execution, and the task instance will be returned to the task pool to await subsequent scheduling. If no higher-priority task instance is waiting for resources, the task instance will be sent to the corresponding task executor for execution.

[0122] In the technical solution of this application embodiment, by determining whether there are higher priority task instances waiting for resources, the resource allocation and scheduling strategy can be effectively optimized, ensuring that high priority task instances can be executed first, thereby improving the efficiency and responsiveness of task instance execution.

[0123] In another example, for each scheduled task instance, when executing the task instance, it may include: sending the task instance to the task executor corresponding to the task type for execution based on the task type of the task instance.

[0124] The task type includes either data computation (such as calculations through a model) or data cleaning. If the task instance includes data processing logic statements, these statements are sent to the corresponding task executor for execution during the execution of the task instance.

[0125] For example, the dependencies between task instances in the target task flow graph are first analyzed. These dependencies describe the order and conditions of task instances. For instance, task instance 'a' can only begin execution after its parent task instance 'b' has completed. For each scheduled task instance, after confirming that all parent task instances it depends on have completed execution and the task instance is sent to the corresponding task executor, the task type of the task instance is checked. The task type can include data computation or data cleaning. Based on the task type, the appropriate task executor is selected to process the task instance. If the task instance includes data processing logic statements, these statements must first be sent to the corresponding task executor during execution. Then, based on the output data of the task executor corresponding to the data processing logic statement, the task instance is executed by the task executor corresponding to its task type.

[0126] In the technical solution of this application embodiment, a suitable task executor can be intelligently selected for execution based on the task type of the task instance. This not only improves the efficiency of task instance processing but also ensures that each type of task instance is executed by a dedicated task executor, thereby fully utilizing the specific capabilities and advantages of the task executor.

[0127] For example, for each scheduled task instance, when executing the task instance, if the task type of the task instance is data computation, the task instance is sent to the first task executor so that the first task executor can call the computation model to process the task instance.

[0128] For example, for each scheduled task instance, when executing the task instance, it is first checked whether the task type of the task instance is a data computation type. If it is determined that the task type of the task instance is a data computation type, the task instance is sent to a first task executor that has the ability to invoke the corresponding computation model. The first task executor processes the task instance by invoking the computation model.

[0129] For example, the first task executor could be Tritton, used to execute computational model inference services. Computational task instances are submitted to the Tritton platform for execution. See also... Figure 1b When a task instance is detected to be of the data computation type, the task scheduling system submits the computation-type task instance, i.e., the artificial intelligence task instance (AI task instance), to the first task executor platform for execution. Additionally, the task scheduling system can directly query the Tritton platform to monitor the execution status of the AI ​​task instance.

[0130] In the technical solution of this application embodiment, the task instance is processed by calling the computing model through the first task executor, thereby improving the calling efficiency and accuracy of the computing model.

[0131] For example, for each scheduled task instance, when executing the task instance, if the task type of the task instance is data cleaning, the task instance is sent to the second task executor so that the data indicated by the data source node can be obtained through the second task executor and the data can be cleaned.

[0132] For example, for each scheduled task instance, upon execution, the system first checks whether the task type is data cleaning. If the task type is determined to be data cleaning, the task instance can be sent to the second task executor. The second task executor, based on the task instance requirements and data cleaning process, retrieves the data to be cleaned from a specified data source using a suitable data acquisition mechanism (e.g., accessing a Hive database). After retrieving the data, the second task executor performs data cleaning on the data according to the task instance's runtime parameters.

[0133] For example, the second task executor could be Spark, and a data cleaning task instance is a Spark-type task instance that needs to be run by Spark. Spark task instances run Spark code packages. See also... Figure 1bThe task scheduling system passes the data cleaning task instance to the second task executor for execution. It's worth noting that data can be retrieved from data warehouse tools for cleaning during the data cleaning process. Furthermore, depending on the data volume, data characteristics, and specific application scenario, storage systems such as MySQL, ClickHouse, HBase, or Minio can be chosen to replace the Hive database. Additionally, the third task executor stores the status of all types of Spark task instances. The task scheduling system queries the corresponding workflow instance in the third task executor to understand and update the status of the tasks processed by the second task executor.

[0134] In the technical solution of this application embodiment, when the task type of the task instance is data cleaning, the task instance is sent to the second task executor so that the second task executor can obtain the data indicated by the data source node and perform data cleaning on the data. Data cleaning is usually a necessary operation, and the data cleaning type and other data processing types are usually not strongly related. Therefore, this pre-processing type of data cleaning can be generated into a separate task instance to improve its versatility. Moreover, data cleaning can improve the efficiency and effectiveness of subsequent task instance processing and calculation.

[0135] For example, if the task instance includes data processing logic statements, during the execution of the task instance, the data processing logic statements are sent to a third task executor for execution.

[0136] For example, for each scheduled task instance, when executing the task instance, it is first detected whether the task type of the scheduled task instance includes data processing logic statements. If it is determined that the task instance includes data processing logic statements, the data processing logic statements are sent to the third task executor during the execution of the task instance. After receiving the data processing logic statements, the third task executor will execute them using its data processing capabilities.

[0137] For example, the third task executor could be DolpinScheduler(DS). See also Figure 1b When the task scheduling system needs to execute data processing logic statements, it sends these statements to a third-party task executor. The third-party task executor then retrieves the relevant data by querying a data warehouse tool.

[0138] In the technical solution of this application embodiment, if the task instance includes data processing logic statements, during the execution of the task instance, the data processing logic statements are sent to a third task executor so that they can be executed by the third task executor, thereby improving the efficiency and performance of the system.

[0139] For example, for each scheduled task instance, when executing the task instance, if the task type of the task instance is data cleaning, the task instance is sent to the third task executor so that the third task executor can forward the task instance to the second task executor.

[0140] For example, for each scheduled task instance, when executing the task instance, the system first checks whether the task type is data cleaning. If the task type is determined to be data cleaning, the task instance is sent to a third task executor. The third task executor can help the system optimize resource utilization and adjust the execution path and scheduling strategy of the task instance according to requirements. Therefore, after receiving the task instance, the third task executor uses its resource optimization capabilities to flexibly adjust the execution path and scheduling strategy of the task instance according to the system load and task instance priority. The third task executor also forwards the task instance to the second task executor and updates the status of the task instance.

[0141] For example, the third task executor can be DolpinScheduler(DS). The second task executor can be Spark. When the task type of the task instance is data cleaning, the task instance is a Spark task instance. See also... Figure 1b The data cleaning task instance is sent to a third task executor, which then triggers the data cleaning task instance, allowing it to be submitted to a second task executor for execution. It's worth noting that during the data cleaning process, the task instance can retrieve data from data warehouse tools for cleaning. Furthermore, depending on the data volume, data characteristics, and specific application scenario, storage systems such as MySQL, ClickHouse, HBase, or Minio can be chosen to replace the Hive database.

[0142] In the technical solution of this application embodiment, when the task type of the task instance is data cleaning, the task instance is sent to the third task executor so that the task instance can be forwarded to the second task executor through the third task executor, thereby enhancing the scalability and flexibility of the system.

[0143] In another example, for each scheduled task instance, when executing the scheduled task instance, if there are no parent task instances that the scheduled task instance depends on, or if all parent task instances that the scheduled task instance depends on have been executed, the scheduled task instance is sent to the task executor corresponding to the task type for execution.

[0144] For example, for each scheduled task instance, when executing the scheduled task instance, after determining the scheduled task instance, based on the dependencies between task instances, it can be determined whether there is a parent task instance that the scheduled task instance depends on. If there is no parent task instance that the scheduled task instance depends on, the scheduled task instance can be directly sent to the task executor corresponding to the task type. If there is a parent task instance that the scheduled task instance depends on, it is necessary to determine whether all parent task instances that the scheduled task instance depends on have been executed. If all parent task instances that the scheduled task instance depends on have been executed, it means that the scheduled task instance meets the execution conditions, and the scheduled task instance can be sent to the task executor corresponding to the task type. Alternatively, if not all parent task instances that the scheduled task instance depends on have been executed, the scheduled task instance continues to wait.

[0145] In another example, parsing the target data flow graph may include: for any node among multiple nodes in the target data flow graph, before parsing the node to obtain the corresponding task instance or converting the node into a data processing logic statement, traversing all the parent nodes of the node in a breadth-first manner, and after parsing all the parent nodes to obtain the output data of all the parent nodes, using the output data as the input data of the node and parsing the node.

[0146] For example, the target data flow graph is a directed graph where nodes represent task instances and edges represent dependencies between task instances. Except for data source nodes, each node can only depend on its parent node to determine its input data; that is, only the output data of the parent node can be used as the input data for the current node. The nodes in the target data flow graph can be divided into multiple levels, and breadth-first traversal processes nodes hierarchically. That is, the nodes of the first level are parsed first, then the second level, and so on, until the last level. Therefore, breadth-first traversal ensures that all parent nodes (i.e., the parent nodes that the current node depends on) have been parsed before the current node is parsed, thus obtaining all the necessary input data for the current node.

[0147] For any node in the target data flow graph, before parsing the node to obtain its corresponding task instance or converting it into a data processing logic statement, a breadth-first traversal is used to identify and determine all the parent nodes that the node depends on, based on the dependencies between nodes in the target data flow graph. Then, all the parent nodes are parsed to obtain the output data of each parent node. The output data of all the parent nodes that the node depends on serves as the input data for that node. Only after all the input data for a node has been determined can the node be parsed.

[0148] For example, the node to be parsed is a data JOIN node, which requires two input data points. Therefore, its input count depends on the two parent nodes that the data JOIN node depends on. Please refer to [link / reference]. Figure 5 Before parsing the data JOIN node, a breadth-first traversal is used to determine the two parent nodes that the data JOIN node depends on: the data source 1 node and the data cleaning node. Then, the data source 1 node and the data cleaning node are parsed to determine their corresponding output data. The output data of the data source 1 node and the data cleaning node are used as the input data for the data JOIN node, and the data JOIN node is then parsed.

[0149] In the technical solution of this application embodiment, for any node among multiple nodes in the target data flow graph, before parsing the node to obtain the corresponding task instance or converting the node into a data processing logic statement, all the parent nodes of the node are traversed in a breadth-first traversal manner. After parsing all the parent nodes to obtain the output data of all the parent nodes, the output data is used as the input data of the node and the node is parsed. This ensures that the output data of all the parent nodes that the node depends on are fully collected, thereby avoiding the omission of parent data and ensuring that the node is parsed correctly.

[0150] Figure 6 This is a schematic diagram of the process for obtaining the target data flow graph provided in an embodiment of this application.

[0151] like Figure 6 As shown, the method 600 for obtaining a target data flow graph provided in this application includes steps S610-S620.

[0152] Step S610: Obtain the workflow.

[0153] Step S620: Based on the workflow, construct the target data flow diagram.

[0154] The workflow includes target node information, edge information between target nodes, and configuration information of target nodes.

[0155] For example, the target node information, edge information between target nodes, and configuration information of target nodes are retrieved from the database storing the workflow through a query. Then, the target node information, edge information between target nodes, and configuration information are parsed, and the target data flow graph is generated based on the parsed information. It should be noted that at this stage, no specific task instances have been instantiated.

[0156] In the technical solution of this application embodiment, a general target data flow graph is constructed based on the workflow, which is flexible and convenient.

[0157] Figure 7 This is a schematic diagram of the process for obtaining the target data flow graph provided in an embodiment of this application.

[0158] like Figure 7 As shown, the method 700 for obtaining a target data flow graph provided in this application includes steps S710-S730.

[0159] Step S710: Output the candidate nodes to the orchestration interface.

[0160] In step S720, the orchestration operation for the candidate node is received through the orchestration interface, and the information of the target node, the edge information between the nodes, and the configuration information of the target node are obtained.

[0161] Step S730: Based on the target node's information, edge information, and configuration information, the workflow is obtained.

[0162] For example, a list of candidate nodes is displayed on the orchestration interface. Candidate nodes may include data source nodes, data processing nodes, data filtering nodes, data connection nodes, data combination nodes, and operator nodes. Data processing nodes include data cleaning nodes and data synchronization nodes. Based on specific circumstances and actual needs, the required nodes are selected from the candidate nodes, and connection and configuration operations are performed on them on the orchestration interface to achieve orchestration operations for the candidate nodes. The orchestration interface receives orchestration operations for candidate nodes, obtaining information about the target node, edge information between nodes, and configuration information of the target node. Then, the target node information, edge information between nodes, and configuration information of the target node are submitted. This target node information, edge information between nodes, and configuration information form a workflow and are stored in the database.

[0163] For example, after completing the orchestration operation for candidate nodes through the orchestration interface, the user clicks the "Submit" button on the page to store the target node information, the edge information between nodes, and the target node configuration information into the MySQL database, thereby completing the workflow creation process.

[0164] In the technical solution of this application embodiment, candidate nodes are output to the orchestration interface. The orchestration interface receives orchestration operations for the candidate nodes, obtaining information about the target node, edge information between nodes, and configuration information of the target node. This process is simple to operate and reduces development difficulty. Based on the target node information, edge information, and configuration information, a workflow is obtained, and a universal target data flow graph is constructed. This process is flexible and convenient.

[0165] Figure 8a This is a schematic diagram illustrating the process of obtaining node information provided in an embodiment of this application.

[0166] like Figure 8a As shown, the node information acquisition method 800 provided in this application includes steps S810-S840.

[0167] In step S810, a selection operation is received through the orchestration interface, and a target node is selected from the candidate nodes based on the selection operation to obtain the information of the target node.

[0168] Step S820: Output the target node to the orchestration interface.

[0169] In step S830, the connection operation performed on the target node is received through the orchestration interface, and the edges between the target nodes are generated based on the connection operation to obtain the edge information.

[0170] The target node includes at least one of the following: data source node, data processing node, and operator node.

[0171] For example, candidate nodes include data source nodes, data processing nodes, and operator nodes. Data processing nodes include data cleaning nodes, data filtering nodes, data connection nodes, and data combination nodes. The data source includes at least one data source. The operator node includes at least one model. The user selects a target node from the above nodes based on the actual situation and needs by performing selection operations (such as clicking or dragging) through the orchestration interface. Based on the user's selection operation, the node selected from the candidate nodes is used as the target node, and the target node's information (such as the target node's name and configuration parameters) is obtained through the target node. The obtained target node is output on the orchestration interface so that the user can view it and perform further operations. On the orchestration interface, if the target node is not a data source node, at least one of the remaining target nodes already selected on the orchestration interface can be selected as the input node of the current target node, depending on the specific situation of the target node. The user will connect the input node determined to be connected to the target node with the target node. The connection operation performed on the target node is received through the orchestration interface, and an edge between the target node and the input node is generated based on the connection operation. Based on the edges generated between the target node and the input node, information such as the dependencies between nodes and the data flow can be represented, thus obtaining edge information.

[0172] For example, operator nodes support various mechanistic models and AI models, represented as models. See also... Figure 8b The orchestration interface mainly consists of an operator component library and a canvas. The operator component library includes a component dropdown list, with components categorized into three types: data source, data processing, and model. The data processing components include four parts: data cleaning, data filtering, data joining (JOIN), and data combination (NUION). Users enter the orchestration interface and select the desired component as the target node from the dropdown list.

[0173] If a user needs to clean the data source, they drag a data cleaning node as the target node and connect it to the data source node that needs cleaning. If a user needs to process the output data of a node (data filtering, data joining, data combination), they drag the corresponding type of data processing node to the canvas as the target node and connect it to the node that needs processing. The parent node that a data processing node depends on can be any node that can produce data output, such as data source nodes, data cleaning nodes, data filtering nodes, data joining nodes, data combination nodes, and operator nodes. Users can also drag a model (operator) to the canvas as the target node and connect its data processing node.

[0174] It's important to note that the data source node identifies the original data source of this workflow, representing a real, existing database table. Data source nodes support mainstream databases such as Hive, ClickHouse, and MySQL. The model supports various mechanistic models and AI models.

[0175] It is important to emphasize that while there can be multiple nodes, a node without a parent node is always a data source node, and the parent node that a data cleaning node depends on can only be a data source node. The input to data join nodes and data combination nodes comes from two nodes.

[0176] For example, please see Figure 8c The system is configured with three data source nodes: Data Source 1, Data Source 2, and Data Source 3. The output data from Data Source 2 requires cleaning and processing; therefore, the data cleaning node is connected to Data Source 2 as the target node. The input data for Model 1 comes directly from Data Source 3. The output data from Data Source 1 needs to be joined with the cleaned output data from Data Source 2; therefore, the data union node is connected to the data source node and the data cleaning node as the target node. The input data for Model 3 comes from the output data of the data union node. The output data from the data cleaning node needs to be filtered; therefore, the data filtering node is connected to the data cleaning node as the target node. The output data from the data filtering node needs to be joined with the output data of Model 1; therefore, the data union node is connected to Model 1 as the target node and the data filtering node as the target node. The output data from Model 3 needs to be joined with the output data of the data union node; therefore, the data union node is connected to Model 3 as the target node and the data union node as the data union node. Model 2's input data comes from data source node 2, which has been cleaned and filtered before being joined with the result of Model 1. Model 4's input data comes from data source node 2, which has been cleaned and filtered before being joined with the result of Model 1, and then joined again with the output of Model 3. In this process... Figure 8c In this context, the data source node only contains output data, while all other nodes contain both input and output data.

[0177] Step S840: Receive configuration operations for the target node through the orchestration interface, and generate configuration information for the target node based on the configuration operations.

[0178] For example, after determining the target node, it needs to be configured so that instantiation can be completed later. The user performs configuration operations on the target node on the orchestration interface. The orchestration interface receives the configuration operations for the target node and determines the configuration items included in the target node. Configuration information for the target node is generated based on the received configuration items.

[0179] For example, when the target node is a data cleaning node, the user can open the data cleaning node configuration page by clicking "Edit". The user can then configure the specific data cleaning rules on this page. Typically, data cleaning nodes have pre-configured general cleaning rules, so users do not need to configure the rules again when dragging the data cleaning node onto the canvas. This design effectively reduces the number of steps required from the user, making the entire process more efficient and convenient.

[0180] Users can open the data filter node configuration page by clicking "Edit". Users can configure multiple filter conditions on this page; the filter fields are derived from the output data of the parent node that the data filter node depends on.

[0181] In one example, by querying the parent node upon which the data filtering node depends, we can determine that the parent node outputs three fields: a, b, and c. The data filtering node connected to this parent node allows filtering of these three fields based on specific conditions; for example, the condition a > 100 can be set. After filtering by the data filtering node, the data passed to subsequent nodes only contains data that meets the condition, i.e., data where the value of field a is greater than 100.

[0182] A JOIN node requires both parent nodes to be joined as data sources. Users can access the JOIN node configuration page by clicking "Edit." On this page, users can specify the left and right data source fields, the connection fields, and rename them to meet the input requirements of subsequent nodes.

[0183] In one example, the workflow of a JOIN node can be as follows: A JOIN node needs to connect two data source nodes, namely data source A and data source B. Fields a, b, and c are selected from the output data of data source A, and fields m, n, t, and y are selected from the output data of data source B. The join condition is determined by the ON condition in the JOIN statement, for example, selecting fields a and b from data source A to join fields t and y from data source B. This means that only when a = t and b = y will the data from data source A and data source B be joined into a single row as output. The joined row will contain fields a, b, c, m, n, t, and y, the contents of which come from data rows in data source A and data source B that meet the join condition.

[0184] In the technical solution of this application embodiment, the orchestration interface provides a visual interface, on which users can quickly select target nodes and perform connection and configuration operations to obtain target node information, edge information, and target node configuration information. This process is simple to operate, reduces development difficulty, and thus improves the efficiency of workflow construction. Users can manage various types of nodes in an intuitive way, realizing batch scheduling of nodes and full lifecycle management of task instances.

[0185] For example, please refer to Figure 9 , Figure 9 To obtain the workflow flowchart.

[0186] For example, when a user enters the DAG orchestration page, a dropdown display component and a model list are displayed. The user can drag one or more data source components from the dropdown display component onto the canvas. Then, the user determines whether the data source needs data cleaning. If the data source needs data cleaning, the user drags a data cleaning component from the dropdown display component onto the canvas. Then, the user connects the nodes using connection lines, linking the data source node that needs data cleaning to the data cleaning node. Next, the user right-clicks to open the configuration interface to configure the cleaning rules and clicks the save button on the configuration page to save the current node configuration. After data cleaning is completed, or if the data source does not need data cleaning, the user also needs to determine whether data processing is required. If no data processing is required, the user drags one or more model nodes from the model list onto the canvas. If data processing is required, the user drags a data processing node from the dropdown display component onto the canvas. For example, when the data processing node is a data filtering node, it is connected to the preceding data source node, and then the user right-clicks to open the data processing configuration page to configure the data filtering conditions. When the data processing node is a JOIN node, it needs to be connected to the data source nodes of the left and right tables, and the data fields, field aliases, and JOIN fields of the left and right tables need to be configured. When the data processing node is a NUION node, it also needs to be connected to the data source nodes of the left and right tables, and the data fields and field aliases of the left and right tables need to be configured. After configuring the data processing node, drag one or more model nodes from the model list to the canvas, and connect the model to the preceding data source node using connectors. Then, right-click to open the model configuration page, select the model version, configure the model input fields, and configure the model running parameters. Then determine whether to add more nodes. If no more nodes need to be added, click Submit to get the workflow. If more nodes need to be added, determine again whether data processing is needed, and then repeat the above operations.

[0187] After obtaining the workflow, please refer to Figure 10The first step is to create a task instance. When a message or event triggering workflow execution is received, the system first queries the storage system for workflow information (including nodes and edges), and then generates a Directed Acyclic Graph (DAG) (i.e., the target data flow graph) based on the workflow information. Next, it determines whether this trigger only requires executing a portion of the DAG; if so, it extracts the necessary portion. After obtaining the DAG to be executed, the system verifies the status of each node. Once all nodes are in a configured complete state, task instance generation can begin.

[0188] When generating task instances, a width traversal method is used to iterate through each DAG node in turn, corresponding to a current node. The type of the current node determines whether it needs to be instantiated. If it does, a task instance corresponding to the current node is generated. If the current node does not need to be instantiated, the output information of the current node (table information, field information, filtering conditions, etc.) is directly pushed down to other nodes in the next layer that depend on the current node. For current nodes that do not need to be instantiated, such as data join nodes, data combination nodes, data filtering nodes, etc., only the data query information corresponding to the current node needs to be pushed down.

[0189] For the current node that needs to be instantiated, when generating a task instance, the relevant tables, fields, and filtering information are output based on the data pushed down from the parent node to generate a query SQL for retrieving task data. Simultaneously, the relevant runtime parameter information is obtained from the node configuration. Finally, the query SQL for retrieving task data, runtime parameters, result storage location, and other information corresponding to the task are stored in the task instance parameters. During traversal, dependency information (edges) for the task instance is generated based on the DAG dependencies. After sequential traversal, a task instance DAG (i.e., the target task flow graph) is obtained, and this task instance DAG (i.e., the target task flow graph) is stored in the storage system (MySQL) to await task scheduling by the scheduling system.

[0190] The second step is to schedule the execution of task instances. For example, task instances may be triggered externally at a set time. When a task instance is triggered, the system first queries the database storing task instances and their dependencies for successfully submitted task instances. Then, it checks whether all the parent task instances that the task instance depends on have been executed. If all parent task instances have been executed, it checks whether there are sufficient idle resources in the resource queue of the task executor corresponding to the task instance. If there are sufficient idle resources, it checks whether there are any higher-priority task instances waiting for resources. If there are no higher-priority task instances waiting for resources, the task instance is distributed to the corresponding task executor for execution. If not all parent task instances have been executed, or there are no sufficient idle resources, or there are higher-priority task instances waiting for resources, the task instance is placed in the task pool to wait, and then the above operations are repeated.

[0191] This application provides a task scheduling and execution device 1100. Please refer to [link to relevant documentation]. Figure 11 The task scheduling and execution device 1100 includes:

[0192] The acquisition module 1110 is used to acquire a target data flow graph, wherein the target data flow graph includes multiple nodes, and the multiple nodes include a first type of node;

[0193] The parsing module 1120 is used to parse the target data flow graph and generate task instances corresponding to the first type of nodes;

[0194] The module 1130 is used to generate the dependency relationship between task instances based on the dependency relationship between the first type of nodes in the target data flow graph, and to obtain the target task flow graph based on the dependency relationship between the task instances.

[0195] The scheduling and execution module 1140 is used to schedule and execute tasks based on the target task flow graph.

[0196] It is understood that for a detailed description of the task scheduling and execution device 1100, please refer to the description of the task scheduling and execution method above.

[0197] For example, the process involves parsing the target data flow graph to generate data processing logic statements corresponding to the second type of nodes; determining the next level of the first type of nodes based on the dependency relationship between the second type of nodes and the first type of nodes in the target data flow graph; and adding the data processing logic statements to the task instance corresponding to the next level of the first type of nodes. This ensures that when the corresponding task instance is executed during task scheduling and execution based on the target task flow graph, the data processing logic statements are executed first to obtain the input data of the task instance.

[0198] For example, the obtaining module 1130 includes: for two first-type nodes with a dependency relationship in the target data flow graph, if there is no second-type node between the two first-type nodes, determining the dependency relationship between the two first-type nodes as the dependency relationship between the two task instances corresponding to the two first-type nodes; and / or if there is a second-type node between the two first-type nodes, obtaining the direct dependency relationship between the two first-type nodes based on the dependency relationship between the second-type node and the two first-type nodes respectively, and determining the direct dependency relationship as the dependency relationship between the two task instances corresponding to the two first-type nodes.

[0199] For example, the scheduling and execution module 1140 includes: querying task instances in the target task flow graph that are in the successfully created state; responding to the triggering of a task, scheduling and executing each task instance in the successfully created state in sequence according to the dependency relationship between task instances in the target task flow graph; wherein, for each scheduled task instance, if the parent task instance to which the task instance depends has been executed, the task instance is sent to the corresponding task executor for execution; if the parent task instance to which the task instance depends has not been executed, the task instance is placed in the task pool to wait.

[0200] This application provides a task scheduling and execution system, characterized in that the task scheduling and execution system includes a task scheduling system and a task executor, the task scheduling system and the task executor are communicatively connected, and the task scheduling and execution system is capable of executing the steps of the method according to any one of the above claims.

[0201] This application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the above embodiments.

[0202] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.

[0203] One embodiment of this application provides a computer program product including instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the steps of the method described in any of the above embodiments.

[0204] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequential list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this application, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0205] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0206] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for task scheduling and execution, the method comprising: The method comprises: obtaining a target data flow graph, wherein the target data flow graph comprises a plurality of nodes, and the plurality of nodes comprise first-type nodes; parsing the target data flow graph to generate task instances corresponding to the first-type nodes; generating a dependency relationship between the task instances according to a dependency relationship between the first-type nodes in the target data flow graph, and obtaining a target task flow graph based on the dependency relationship between the task instances; performing task scheduling and execution based on the target task flow graph.

2. The method of claim 1, wherein, The plurality of nodes further comprise second-type nodes, and the method further comprises: in the step of parsing the target data flow graph to generate task instances corresponding to the first-type nodes, data processing logic statements corresponding to the second-type nodes are also generated; determining a next first-type node of the second-type node according to a dependency relationship between the second-type node and the first-type nodes in the target data flow graph, and adding the data processing logic statement to a task instance corresponding to the next first-type node, so that the data processing logic statement is executed first to obtain input data of the task instance when the corresponding task instance is executed in the process of performing task scheduling and execution based on the target task flow graph.

3. The method of claim 1, wherein, The generating of the dependency relationship between the task instances according to the dependency relationship between the first-type nodes in the target data flow graph comprises: for two first-type nodes having a dependency relationship in the target data flow graph, if there is no second-type node between the two first-type nodes, determining the dependency relationship between the two first-type nodes as a dependency relationship between two task instances corresponding to the two first-type nodes; and / or if there is a second-type node between the two first-type nodes, obtaining a direct dependency relationship between the two first-type nodes based on the dependency relationship between the second-type node and the two first-type nodes respectively, and determining the direct dependency relationship as a dependency relationship between two task instances corresponding to the two first-type nodes.

4. The method of claim 2, wherein, The performing of task scheduling and execution based on the target task flow graph comprises: scheduling task instances from the target task flow graph according to the dependency relationship between the task instances; for a scheduled task instance, obtaining input data of the task instance based on output data of an upper task instance of the task instance, and executing the task instance.

5. The method of claim 4, wherein, The obtaining of the input data of the task instance based on the output data of the upper task instance of the task instance comprises: for a scheduled task instance, if the task instance comprises a data processing logic statement, processing the output data of the upper task instance based on the data processing logic statement to obtain a processing result; taking the processing result and / or the output data of the upper task instance as the input data of the task instance.

6. The method of claim 1, wherein, The performing of task scheduling and execution based on the target task flow graph comprises: querying task instances in a created-success state in the target task flow graph, and in response to triggering of a task, scheduling and executing the task instances in the created-success state in the target task flow graph in sequence according to the dependency relationship between the task instances in the target task flow graph, If the task instance is successfully executed, the state of the task instance is updated to a success state, and the next task instance dependent on the task instance is executed.

7. The method of claim 6, wherein, The method further includes: If the task instance is successfully executed, the state of the task instance is updated to a success state, and the next task instance dependent on the task instance is executed. If the task instance is successfully executed, the state of the task instance is updated to a success state, and the next task instance dependent on the task instance is executed.

8. The method of claim 6, wherein, The task instance includes at least data input, an output-associated query statement, output result table information, field information, and a running parameter.

9. The method of claim 6, wherein, When the task instance is executed, the task executor executes according to the running parameter of the task instance. When the task instance is executed, the task executor executes according to the running parameter of the task instance. When the task instance is executed, the task executor executes according to the running parameter of the task instance.

10. The method of claim 6, wherein, When the task instance is executed, the task executor executes according to the running parameter of the task instance. When the task instance is executed, the task executor executes according to the running parameter of the task instance. When the task instance is executed, the task executor executes according to the running parameter of the task instance.

11. The method of claim 3, wherein, When the task instance is executed, the task executor executes according to the running parameter of the task instance. When the task instance is executed, the task executor executes according to the running parameter of the task instance.

12. A task scheduling and executing apparatus, characterized by comprising: When the task instance is executed, the task executor executes according to the running parameter of the task instance. When the task instance is executed, the task executor executes according to the running parameter of the task instance. When the task instance is executed, the task executor executes according to the running parameter of the task instance. The parsing of the target data flow graph includes: Before the node is parsed to obtain the task instance corresponding to the node or the node is converted into a data processing logic statement, all nodes at a previous level of the node are traversed in a breadth-first manner, and the output data of all nodes at the previous level is obtained by parsing all nodes at the previous level. The apparatus includes: The obtaining module is configured to obtain a target data flow graph, wherein the target data flow graph includes a plurality of nodes, and the plurality of nodes include first-type nodes. The parsing module is configured to parse the target data flow graph to generate task instances corresponding to the first-type nodes. The obtaining module is configured to generate a dependency relationship between task instances according to a dependency relationship between the first-type nodes in the target data flow graph, and obtain a target task flow graph based on the dependency relationship between the task instances. The scheduling and executing module is configured to perform task scheduling and execution based on the target task flow graph.

13. A task scheduling and execution system, characterized by, The task scheduling and execution system comprises a task scheduling system and a task executor, the task scheduling system is in communication connection with the task executor, and the task scheduling and execution system can execute the steps of the method according to any one of claims 1-11.

14. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1-11.

15. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1-11.

16. A computer program product, comprising instructions therein, characterised in that, The instructions are executed by the processor of the electronic device to enable the electronic device to execute the steps of the method in any one of claims 1-11.

Images