Task scheduling method and device, equipment and storage medium

Abstract

The application relates to a task scheduling method and device, equipment and a storage medium. The method comprises the following steps: obtaining a relationship diagram for reflecting the dependency relationship between tasks from a relationship diagram editing area of a visual configuration interface; analyzing the relationship diagram to generate a configuration file compatible with a task scheduling system; obtaining scheduling information of each task from a scheduling information configuration area of the visual configuration interface after uploading the configuration file to the task scheduling system; and sending the scheduling information to the task scheduling system, so that the task scheduling system performs task scheduling on each task based on the configuration file and the scheduling information. The relationship diagram editing area and the scheduling information configuration area of the visual configuration interface realize configuration of the scheduling information and the dependency relationship used for analyzing the configuration file in one configuration process, and therefore, the configuration process is simple.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a task scheduling method, apparatus, device, and storage medium. Background Technology

[0002] Azkaban, as a workflow scheduling tool, integrates scheduling, orchestration, failure retry, and email alerts. In the current application, users manually write the configuration files for each task and package all configuration files and necessary resource files into a single zip file. This zip file is then uploaded to the Azkaban server through the user interface provided by the server. The user then uses this interface to access the individual task scheduling information configuration pages on the Azkaban server, and configures the scheduling information on each page sequentially by switching between them.

[0003] However, the configuration process is quite complex because the writing of configuration files and the configuration of scheduling information are done by the user in multiple steps. Summary of the Invention

[0004] This application provides a task scheduling method, apparatus, device, and storage medium to solve the problem of complex configuration processes in related technologies.

[0005] Firstly, a task scheduling method is provided, including:

[0006] From the relationship diagram editing area of ​​the visual configuration interface, obtain the relationship diagram that reflects the dependencies between tasks;

[0007] Parse the relationship graph to generate a configuration file adapted to the task scheduling system;

[0008] After the configuration file is uploaded to the task scheduling system, the scheduling information of each task is obtained from the scheduling information configuration area of ​​the visual configuration interface. The scheduling information includes the execution time and alarm information of the project to which each task belongs.

[0009] The scheduling information is sent to the task scheduling system so that the task scheduling system performs task scheduling for each task based on the configuration file and the scheduling information.

[0010] Optionally, the relationship graph is parsed to generate a configuration file adapted to the task scheduling system, including:

[0011] According to a preset computer language format, a relational graph description corresponding to the relational graph is generated. The relational graph description includes a first part of the description for indicating the node information of each node and a second part of the description for indicating the node relationship of each node. Any task has a unique corresponding node in the relational graph.

[0012] Obtain the node information of each node from the first part of the description, and obtain the dependent nodes of each node from the second part of the description;

[0013] Based on the node information of each node and the dependent nodes of each node, a configuration file adapted to the task scheduling system is generated.

[0014] Optionally, a relational diagram description corresponding to the relational diagram is generated according to a preset computer language format, including:

[0015] Identify at least one node in the relationship graph according to the pre-set node attributes;

[0016] Obtain the node information of each node respectively;

[0017] Identify the directed line segments that are connected to each of the nodes;

[0018] Based on the direction of the directed line segment, the dependent nodes of each node are determined respectively;

[0019] The relationship graph description is obtained by describing the node information of each node and the dependent nodes of each node according to the preset computer language format.

[0020] Optionally, the dependent nodes of each node are obtained from the description in the second part, including:

[0021] For any given node, find the target node relationship of the node described in the second part.

[0022] Obtain the starting node from the target node relationship;

[0023] The starting node is determined as the dependent node of the node.

[0024] Optionally, based on the node information of each node and the dependent nodes of each node, a configuration file adapted to the task scheduling system is generated, including:

[0025] Determine the leaf nodes from each of the nodes;

[0026] Create a terminal node that has a dependency relationship with the leaf node, wherein the dependent node of the terminal node is the leaf node;

[0027] Obtain the node information of the last node, and generate a configuration file corresponding to the last node based on the node information of the last node and the dependent nodes of the last node;

[0028] Based on the node information of each node and the dependent nodes of each node, generate a configuration file corresponding to each node.

[0029] The configuration file corresponding to the last node and the configuration file corresponding to each node are determined as the configuration file.

[0030] Optionally, determining leaf nodes from the nodes includes:

[0031] Based on the dependent nodes of each node, determine the non-leaf nodes from each node;

[0032] The node from which the non-leaf nodes are removed is determined as the leaf node.

[0033] Optionally, determining non-leaf nodes from the nodes based on their respective dependent nodes includes:

[0034] Find the node relationships where the target node is not empty from the description in Part II;

[0035] Obtain the starting node from the node relationships where the target node is not empty;

[0036] The starting node is designated as the non-leaf node.

[0037] Second aspect. A task scheduling device is provided, comprising:

[0038] The first acquisition unit is used to acquire a relationship diagram reflecting the dependencies between tasks from the relationship diagram editing area of ​​the visual configuration interface;

[0039] The generation unit is used to parse the relationship graph and generate a configuration file adapted to the task scheduling system;

[0040] The second acquisition unit is used to acquire the scheduling information of each task from the scheduling information configuration area of ​​the visual configuration interface after the configuration file is uploaded to the task scheduling system. The scheduling information includes the execution time and alarm information of the project to which each task belongs.

[0041] The scheduling unit is used to send the scheduling information to the task scheduling system, so that the task scheduling system can perform task scheduling for each task based on the configuration file and the scheduling information.

[0042] Thirdly, an electronic device is provided, comprising: a processor, a memory, and a communication bus, wherein the processor and the memory communicate with each other via the communication bus;

[0043] The memory is used to store computer programs;

[0044] The processor is used to execute the program stored in the memory to implement the task scheduling method described in the first aspect.

[0045] Fourth aspect. A computer-readable storage medium is provided, storing a computer program that, when executed by a processor, implements the task scheduling method described in the first aspect.

[0046] Compared with the prior art, the technical solution provided in this application has the following advantages: The technical solution provided in this application obtains a relationship diagram reflecting the dependencies between tasks through the relationship diagram editing area of ​​the visual configuration interface, parses the relationship diagram, generates a configuration file adapted to the task scheduling system, and after the configuration file is uploaded to the task scheduling system, obtains the scheduling information of each task through the scheduling information configuration area of ​​the visual configuration interface, and sends the scheduling information to the task scheduling system, so that the task scheduling system performs task scheduling for each task based on the configuration file and the scheduling information. As can be seen from the above, this application embodiment realizes the configuration of scheduling information and the dependencies used to parse the configuration file in a single configuration process through the relationship diagram editing area and scheduling information configuration area of ​​the visual configuration interface. Therefore, the configuration process is simple, which greatly improves the efficiency of users publishing scheduling tasks and shortens the offline data development process.

[0047] In addition, by configuring scheduling information for multiple tasks through a unified visual configuration interface, the need to switch back and forth between multiple pages on Azkaban and the complexity of configuration operations are reduced.

[0048] Furthermore, since the relationship diagram reflecting the dependencies of each task can be configured in the relationship diagram editing area on the visual configuration interface, and the configuration file can be obtained by parsing the relationship diagram, the step of manually writing multiple configuration files can be avoided, which greatly reduces the time cost of task orchestration and reduces or even eliminates the number of trial and error attempts in task orchestration.

[0049] Finally, by linking online development tasks, this solution can meet users' needs for immediate release upon completion of development, providing a one-stop work experience for development and release. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the structure of an Azkaban-based task scheduling system provided in an embodiment of this application;

[0053] Figure 2 A relationship diagram between tasks provided in an embodiment of this application;

[0054] Figure 3 A flowchart illustrating a task scheduling method provided in an embodiment of this application;

[0055] Figure 4 Another flowchart illustrating the task scheduling method provided in this application embodiment;

[0056] Figure 5 Another relationship diagram provided for embodiments of this application;

[0057] Figure 6 This is a schematic diagram illustrating the node relationships shown in an embodiment of this application;

[0058] Figure 7 This is another schematic diagram illustrating the node relationships shown in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the structure of the task scheduling device provided in the embodiments of this application;

[0060] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] To facilitate understanding of the embodiments of this application, some terms involved in this application will be explained below.

[0064] Task scheduling: In computer science, scheduling is a method of allocating tasks to resources for execution. A task can be at least one task in a project or a workflow, including shell scripts, Java programs, MapReduce programs, Hive scripts, etc. Tasks have temporal sequences and dependencies, and they may repeat periodically. Defining execution rules for all tasks and then arranging tasks according to those rules is what task scheduling is all about. Common task scheduling systems include Azkaban, Ooize, and Cascading. The most basic functions of a task scheduling system are task definition and task orchestration. Task definition mainly determines the logic and rules for data computation and processing, including the frequency of task execution, specific execution time, corresponding execution scripts, and parameters. Task orchestration mainly determines the sequence of different tasks, ensuring that tasks are executed in an orderly and efficient manner. The output of task orchestration is a Directed Acyclic Graph (DAG), which allows users to view task dependencies and execution status, providing a visual tracking of the program's execution process.

[0065] Directed Acyclic Graph (DAG): In mathematics and computer science, a directed acyclic graph is a directed graph without loops. The relationships between nodes in a DAG are directional and unidirectional; one can only move from one node to another, and the reverse is impossible. A path can start from any node and traverse multiple nodes along a direction, but it is impossible to return to the starting point. The path traversed cannot form a cycle.

[0066] Leaf node: A node in a tree that has no child nodes is called a leaf node, or simply "leaf", and is also called a terminal node.

[0067] Project release refers to the process of generating a scheduled project on the task scheduling system after the project with the scheduling information and task dependencies configured on the offline data development platform has been configured.

[0068] The application scenarios involved in this application are described below.

[0069] With the development of information technology, data analysis and processing tasks often involve multiple data processing steps, such as data collection, transmission, computation, and visualization. Each step's data processing algorithm needs to be submitted to a computing framework for execution. Some steps can be executed concurrently, while others require dependencies. When tasks are simple, this can be controlled manually. However, when there are many tasks and complex relationships, without a clear task planning graph, it's easy for closed loops to form between tasks, leading to errors, or for multiple parallelizable tasks to fail to execute concurrently, wasting resources. Furthermore, some tasks need to be executed at a specific time, while others have a certain periodicity. To effectively organize such complex execution plans and schedule these complex tasks to run on a distributed computing framework, many task scheduling tools have emerged.

[0070] Azkaban is one of the mainstream task scheduling tools, used for task scheduling and monitoring, including managing program scripts, configuring task dependencies, checking program execution correctness, issuing alerts and retrying when errors occur. It's important to understand that Azkaban is not limited to big data; it can be used for any task scheduling-related task. For example, a simple scheduled email sending task can be scheduled to run on Azkaban by setting the execution time. Within the Azkaban framework, tasks are managed in the order of project, flow, and job. A project contains one or more workflows, and a workflow contains multiple tasks. A job is a process running in Azkaban; it can be a simple Linux command, a Java program, or a complex shell script, MapReduce program, Hive script, or Python script. A job can depend on the results of another job; these dependencies form a workflow, where a group of tasks runs in a specific order.

[0071] Figure 1This application provides an Azkaban-based task scheduling system 100, which may include a management server 101 (Azkaban Web Server), an execution server 102 (AzkabanExecutorServer), and a relational database 103. The management server 101, execution server 102, and relational database 103 can be connected via a network, which can be a wired network, a wireless network, or a combination of both.

[0072] The relational database 103 currently only supports MySQL. The Azkaban database needs to be created and initialized on the MySQL server. The Azkaban system stores configuration file information and most of its state information in the MySQL database 103. Both the management server 101 and the execution server 102 need to access MySQL.

[0073] As the primary administrator of Azkaban, management server 101 provides functions including user login authentication, project creation, project management, task uploading, task scheduling, viewing task execution status, and viewing historical tasks. Users can access management server 101 through browser 104 and perform the above management operations on the user interface (UI) provided by the management server.

[0074] The execution server 102 is the node that actually runs tasks in the entire task scheduling system. It is mainly responsible for submitting and executing workflows, including scheduling Hadoop tasks, scheduling shell script tasks, scheduling Hive tasks, single point of failure, etc., and coordinates the execution of each task through the MySQL database 103.

[0075] In other words, Azkaban's management server 101 acts as a distributor. The management server 101 distributes tasks to the execution server 102. The execution server 102 then retrieves the compressed files uploaded by users from the project files (project_files) in the Azkaban database 103, decompresses them into the local project (projects) folder, and finally submits the tasks to the thread pool. The essence of its execution is to place each job in the thread pool for execution.

[0076] Azkaban defines a key-value (KV) file (properties) format to establish dependencies between tasks and provides an easy-to-use web user interface for maintaining and tracking workflows. First, users need to create a file with the `.job` extension; one `.job` file represents one task. All tasks require a type that guides how they are executed. Azkaban's default task types include `command`, `java`, etc. After defining the job type, the necessary parameters and values ​​are added to this file. One type of parameter that can be added is the `dependencies` parameter, which defines the files that the job depends on. The value is the filename of the dependent file; multiple targets are separated by commas, without extensions. Saving as a `.job` file creates a job. Adding the `dependencies` parameter configures the dependencies between tasks to form a workflow. Finally, all `.job` files and necessary resource files (such as Java packages, Hive scripts, MapReduce JAR files, etc.) are packaged into a zip file and uploaded through the Azkaban user interface. Azkaban decompresses the uploaded zip file and analyzes it to form a directed acyclic graph (DAG) of nodes. This DAG is presented as a dependency graph in the web user interface, allowing users to view the dependencies between tasks and the execution status of each task. Solid lines connect nodes with dependencies in the dependency graph. By default, gray indicates a task is not executed, blue indicates it is executing, green indicates it has succeeded, and red indicates it has failed. Users can then choose to configure scheduled execution or immediate execution. If scheduled execution is selected, at the scheduled time, execution server 102 reads the configuration file from the Azkaban database 103 and downloads the necessary data locally. Execution server 102 then begins executing the workflow, continuously adding the execution status information of each task to database 103. This execution status information can be viewed through the web management server 101.

[0077] For example, Figure 2An example relationship diagram between tasks is provided. The user creates a new file A.job locally, then a file B.job, and adds the statement "dependencies=A" to B.job, defining that task B depends on the execution result of task A. Similarly, a file C.job is created, making task C dependent on task A; then a file D.job is created, making task D dependent on both tasks B and C; finally, a file E.job is created, making task E dependent on task D. After saving the five job files, these five job files and the resource files required by each task are packaged into a zip file (i.e., configuration file). A new project is created in the Azkaban web management interface, including filling in the workflow name, remarks, etc., and then the zip file is uploaded. Thus, a workflow is created. The task scheduling system stores the configuration file in the Azkaban database 103, ultimately presenting a workflow as shown below. Figure 2 The diagram shows the dependencies between tasks.

[0078] In related technologies, users manually write configuration files for each task and package all configuration files and necessary resource files into a zip file. This zip file is then uploaded to the Azkaban server via its user interface. The user then uses this interface to access the scheduling information configuration pages for each task on the Azkaban server, and configures the scheduling information on each page sequentially by switching between them. However, because the configuration files and scheduling information are written and configured by the user in multiple steps, the configuration process is quite complex.

[0079] To address the aforementioned technical problems in related technologies, embodiments of this application provide a task scheduling method, which can be applied to tasks that can schedule tasks with... Figure 1 The electronic device used for communication with the task scheduling system shown in the diagram has an offline data development platform deployed therein, which can present a visual configuration interface. When a project needs to be released, the user starts and runs the offline data development platform and performs scheduling-related configurations from the visual configuration interface provided by the offline data development platform.

[0080] like Figure 3 As shown, the method may include the following steps:

[0081] Step 301: Obtain the relationship diagram reflecting the dependencies between tasks from the relationship diagram editing area of ​​the visual configuration interface.

[0082] The relationship diagram editing area is used to generate a relationship diagram reflecting the dependencies between tasks based on the user's drag-and-drop operations. This area includes a directory tree area and a relationship diagram generation area. The directory tree area includes at least one node, and each node uniquely corresponds to a task. In this embodiment, the nodes in the directory tree area can be set by project, meaning the directory tree area includes the nodes corresponding to each task under this project. In addition, the directory tree area can also include nodes corresponding to tasks under other projects. Of course, to ensure the versatility of nodes, the nodes in the directory tree area can also be categorized so that tasks of the same category under different projects can be represented by the same node. Furthermore, nodes in the directory tree area can be set according to other attributes or parameters; this embodiment does not specifically limit this.

[0083] When generating a relationship graph based on user drag-and-drop operations, the system retrieves at least two nodes dragged by the user from the directory tree area and displays these two nodes in the relationship graph generation area. It also retrieves two nodes indicated by the user from the at least two nodes that can be connected by a directed line segment, and displays the directed line segment between the two nodes according to the user's instructions. The directed line segment represents the dependency relationship between the two nodes. For example, when the first node is connected to the second node by a directed line segment, it indicates that the second node depends on the execution result of the first node.

[0084] In this embodiment, the relationship graph includes, but is not limited to, a DAG graph.

[0085] Step 302: Parse the relationship graph and generate a configuration file adapted to the task scheduling system.

[0086] When an electronic device parses a relationship diagram, it first generates a relationship diagram description corresponding to the relationship diagram according to a preset computer language format. Since the task scheduling system cannot identify the dependencies between tasks through the relationship diagram description in the computer language format, nor can it directly create scheduling tasks based on the relationship diagram description in the computer language format, it is necessary to convert the relationship diagram description into a configuration file that the task scheduling system can recognize.

[0087] In one specific embodiment, in order to facilitate the search of node information and dependent nodes when generating configuration files, the relationship diagram description generated according to a preset computer language format is divided into a first part of the description indicating the node information of each node and a second part of the description indicating the node relationships of each node.

[0088] Optionally, in this embodiment, the preset computer language format includes, but is not limited to, JSON format. When generating a relationship graph description according to JSON format, the relationship graph description is specifically a JSON string conforming to the JSON format. In the JSON string, "nodes" is used to describe the nodes in the relationship graph, and "edges" is used to describe the node relationships between the nodes in the relationship graph.

[0089] The “nodes” section includes the node information of each node, including the node identifier, node name, input port number, output port number, and / or the coordinates of the node in the relationship graph generation area; the “edges” section includes multiple sets of node relationships, each set of node relationships including the node identifier of the starting node (srcNodeId) and the node identifier of the target node (dstNodeId), and a set of node relationships indicates that dstNodeId depends on srcNodeId.

[0090] In this embodiment, the first part describes the “nodes” part of the corresponding JSON string, and the second part describes the “edges” part of the corresponding JSON string.

[0091] Based on the specific implementation of the relationship graph description above, it can be seen that converting a relationship graph into a relationship graph description is essentially the process of electronic devices recognizing relationship graphs. Therefore, as... Figure 4 As shown, this embodiment provides the following steps for generating a relational graph description:

[0092] Step 401: Identify at least one node in the relationship graph according to the preset node attributes;

[0093] Node attributes include, but are not limited to, the node name, the node shape, or the type of the identifier.

[0094] Since directed segments in a relational graph typically do not have names, while nodes are usually named after the task they represent, nodes and directed segments can be distinguished by their names, thus identifying nodes in the relational graph. Alternatively, in a relational graph, nodes are usually not represented as line segments, but as circles, ellipses, or other shapes, so nodes and directed segments can be distinguished by their shapes, thus identifying nodes in the relational graph. Or, in a relational graph, the node identifiers of nodes and the identifiers of directed segments can be represented by different numbers of digits, such as three digits for node identifiers and two digits for directed segment identifiers, thus distinguishing nodes and directed segments based on the type of identifier, thus identifying nodes in the relational graph.

[0095] It should be noted that identifying nodes by node name, node shape, or identifier type is an optional implementation method in this embodiment, and this application does not make specific limitations on it. Of course, other methods in the prior art that can identify nodes in the relationship graph are also suitable for this embodiment.

[0096] Step 402: Obtain the node information of each node separately;

[0097] In one optional implementation, each node in the relationship graph carries node attributes for unique identification, such as node name or node identifier, so the node information of the node can be obtained based on the node attributes carried by the node.

[0098] In this embodiment, each task can be associated with an electronic device in advance, so that the electronic device can obtain relevant information about each task. Therefore, after obtaining the node attributes in the relationship graph, the relevant information of the task corresponding to the node can be obtained according to the node attributes, and the relevant information is determined as the node information of the node.

[0099] Step 403: Identify the directed line segments that are connected to each node;

[0100] Step 404: Based on the direction of the directed line segment, determine the dependent nodes of each node;

[0101] For example, when it is identified that the first node is connected to the second node through a directed line segment, it can be determined that the directed line segment connected to the first node points to the second node, and thus the first node is determined to be the node on which the second node depends.

[0102] Step 405: Describe the node information of each node and the dependent nodes of each node according to the preset computer language format to obtain the relationship graph description.

[0103] Please refer to Figure 5 , Figure 5This is another relational diagram illustrated in this embodiment. In this diagram, each node carries a node name, namely create_table.hql, alter_partition.hql, select.hql, and insert_overwrite_tab, respectively. Therefore, each node in the relational diagram can be identified based on its node name. Furthermore, since the directed line segments connecting the create_table.hql node to the alter_partition.hql node and the select.hql node are respectively, and the directions of the directed line segments are from create_table.hql node to alter_partition.hql node and from create_table.hql node to select.hql node, it can be determined that both alter_partition.hql node and select.hql node depend on the execution result of create_table.hql node. The determination of the node that insert_overwrite_tab node depends on is similar to the aforementioned logic and will not be described further here.

[0104] In one specific implementation, when generating the configuration file based on the above relationship diagram description, specifically, the node information of each node is obtained from the first part of the description, and the dependent nodes of each node are obtained from the second part of the description; and a configuration file adapted to the task scheduling system is generated based on the node information of each node and the dependent nodes of each node.

[0105] Optionally, in this embodiment, the node relationships described in the second part can be expressed as a relative relationship between the starting node and the target node, with the target node depending on the starting node. Therefore, when obtaining the dependent nodes of a node from the second part of the description, the target node relationship is searched in the second part of the description; the starting node in the target node relationship is obtained; and the starting node is determined as the dependent node of the node. It can be understood that the starting node corresponding to the target node described in the second part of the relationship graph is used as the dependent node of that node in the configuration file, where the configuration file is a file recognizable by the task scheduling system.

[0106] Optionally, in this embodiment, to simplify task scheduling complexity and improve scheduling efficiency, a workflow is set up under each project, which includes the tasks described above. For the Azkaban task scheduling system, the naming of workflows in Azkaban is determined by the name of the last node in the dependency relationship. If there are multiple last nodes, the workflow name cannot be determined in advance, which brings certain difficulties to the maintenance of task scheduling. Therefore, in this embodiment, a last node is created based on each node in the above relationship diagram description, and configuration files are generated for each node and the last node respectively.

[0107] In specific implementation, leaf nodes are determined from each node; a terminal node with a dependency relationship to the leaf node is created, and the dependent node of the terminal node is the leaf node; the node information of the terminal node is obtained, and a configuration file corresponding to the terminal node is generated based on the node information and the dependent node of the terminal node; a configuration file corresponding to each node is generated based on the node information and the dependent node of each node; the configuration file corresponding to the terminal node and the configuration files corresponding to each node are determined as the configuration files.

[0108] It should be understood that creating a last node actually includes two parts: determining the dependent nodes of the last node and generating the node information of the last node. Therefore, after the last node is created, the node information of the last node can be obtained so that a configuration file corresponding to the last node can be generated based on the node information and the dependent nodes of the last node.

[0109] The Azkaban configuration file includes three parameters: type, command, and dependencies. `type` represents the node's type, `command` represents the command the node needs to execute, and `dependencies` represents the nodes the node depends on. In this embodiment, `type` is fixed at `command`, indicating that the task corresponding to this node is a shell command task. When determining the value of the `dependencies` parameter for a node, each group of node relationships described in Part II is traversed to find the target node, which is then identified as the target node. The starting node corresponding to the target node is determined as the value of the `dependencies` parameter in the configuration file corresponding to that node. Each "node" in the relationship diagram corresponds to a configuration file, and the node name in the "node" is the configuration file name.

[0110] Please refer to Figure 6 , Figure 6This is a schematic diagram illustrating the node relationships in this embodiment. The diagram includes three sets of node relationships: edge1, edge2, and edge3. Within the two dashed boxes, the node in the left dashed box is the starting node (srcNodeId), and the node in the right dashed box is the target node (dstNodeId). When determining the value of the dependencies parameter in node A's configuration file, all edges with "dstNodeId" set to A are traversed. Three nodes with "srcNodeId" set to B, C, and D are found. Therefore, the value of the dependencies parameter in node A's configuration file is determined to be "B, C, D".

[0111] In this embodiment, when determining leaf nodes from all nodes, since the cost of directly obtaining non-leaf nodes is lower than the cost of obtaining leaf nodes and the computational complexity is smaller, when determining leaf nodes, we first obtain the non-leaf nodes from all nodes, and then remove the non-leaf nodes from all nodes. The remaining nodes are the leaf nodes.

[0112] When determining a leaf node from among the nodes, optionally, the node relationships in the second part of the description where the target node is not empty are searched; the starting node is obtained from the node relationships where the target node is not empty; and the starting node is designated as a non-leaf node.

[0113] In this context, a non-empty target node indicates that the target node does not have a node identifier in the node relationship, meaning that the target node does not actually exist in the relationship graph.

[0114] Please refer to Figure 7 , Figure 7 This is another schematic diagram illustrating the node relationships in this embodiment. The diagram includes four sets of node relationships: edge1, edge2, edge3, and edge4. Within the two dashed boxes in the diagram, the node in the left dashed box is the starting node (srcNodeId), and the node in the right dashed box is the target node (dstNodeId). Figure 7 When identifying a leaf node, traverse all node relationships. If `dstNodeId` is not null, then `srcNodeId` in that relationship is a non-leaf node. Therefore, the traversal process can determine... Figure 7 Nodes A, B, and C are all non-leaf nodes.

[0115] Step 303: After the configuration file is uploaded to the task scheduling system, obtain the scheduling information of each task from the scheduling information configuration area of ​​the visual configuration interface. The scheduling information includes the execution time and alarm information of the project to which each task belongs.

[0116] The alarm information includes, but is not limited to, the duration of each task during execution, the person who notifies the alarm, and / or whether the alarm is sent by telephone.

[0117] Optionally, in other embodiments of this application, the scheduling information may also include information such as project description, number of retries for data dependency failures and interval execution time, number of reruns for a single script and interval execution time.

[0118] The project description can be the name of the project; regarding the number of retries and the interval between executions for data dependency failures: after setting "task dependency" in the script, if the amount of data in the dependent table / partition is 0, the task will automatically fail. Setting a retry mechanism can prevent the task from failing directly due to data delays, and continue running after the dependent data is ready; regarding the number of times a single script is rerun and the interval between executions: if a single script fails, only that script will be re-executed, and scripts that have already been executed successfully will not be re-executed.

[0119] Step 304: Send scheduling information to the task scheduling system so that the task scheduling system can perform task scheduling for each task based on the configuration file and scheduling information.

[0120] The task scheduling system provides a RESTful API interface for interacting with the outside world, so scheduling information can be sent to the task scheduling system through the API interface.

[0121] The technical solution provided in this application embodiment obtains a relationship diagram reflecting the dependencies between tasks through the relationship diagram editing area of ​​the visual configuration interface, parses the relationship diagram, generates a configuration file adapted to the task scheduling system, and after the configuration file is uploaded to the task scheduling system, obtains the scheduling information of each task through the scheduling information configuration area of ​​the visual configuration interface, and sends the scheduling information to the task scheduling system so that the task scheduling system can perform task scheduling based on the configuration file and scheduling information. As can be seen from the above, this application embodiment realizes the configuration of scheduling information and the dependencies used to parse the configuration file in a single configuration process through the relationship diagram editing area and scheduling information configuration area of ​​the visual configuration interface. Therefore, the configuration process is simple, which greatly improves the efficiency of users publishing scheduling tasks and shortens the offline data development process.

[0122] In addition, by configuring scheduling information for multiple tasks through a unified visual configuration interface, the need to switch back and forth between multiple pages on Azkaban and the complexity of configuration operations are reduced.

[0123] Furthermore, since the relationship diagram reflecting the dependencies of each task can be configured in the relationship diagram editing area on the visual configuration interface, and the configuration file can be obtained by parsing the relationship diagram, the step of manually writing multiple configuration files can be avoided, which greatly reduces the time cost of task orchestration and reduces or even eliminates the number of trial and error attempts in task orchestration.

[0124] Finally, by linking online development tasks, this solution can meet users' needs for immediate release upon completion of development, providing a one-stop work experience for development and release.

[0125] Based on the same concept, this application provides a task scheduling device in its embodiments. The specific implementation of this device can be found in the description of the method embodiments section; repeated details will not be repeated here. Figure 8 As shown, the device mainly includes:

[0126] The first acquisition unit 801 is used to acquire a relationship diagram reflecting the dependency relationships between tasks from the relationship diagram editing area of ​​the visual configuration interface;

[0127] The generation unit 802 is used to parse the relationship graph and generate a configuration file adapted to the task scheduling system;

[0128] The second acquisition unit 803 is used to acquire the scheduling information of each task from the scheduling information configuration area of ​​the visual configuration interface after the configuration file is uploaded to the task scheduling system. The scheduling information includes the execution time and alarm information of the project to which each task belongs.

[0129] The scheduling unit 804 is used to send scheduling information to the task scheduling system so that the task scheduling system can perform task scheduling for each task based on the configuration file and the scheduling information.

[0130] In other embodiments of this application, the generation unit 802 is specifically used for:

[0131] According to the preset computer language format, a relational graph description corresponding to the relational graph is generated. The relational graph description includes a first part of the description for indicating the node information of each node and a second part of the description for indicating the node relationship of each node. Any task has a unique corresponding node in the relational graph.

[0132] Obtain the node information of each node from the first part of the description, and obtain the dependent nodes of each node from the second part of the description;

[0133] Based on the node information of each node and the nodes that each node depends on, a configuration file adapted to the task scheduling system is generated.

[0134] In other embodiments of this application, the generation unit 802 is specifically used for:

[0135] Identify at least one node in the relationship graph according to the pre-set node attributes;

[0136] Obtain the node information of each node separately;

[0137] Identify the directed line segments that are connected to each node;

[0138] Based on the direction of the directed line segment, determine the dependent nodes of each node;

[0139] According to the preset computer language format, describe the node information of each node and the dependent nodes of each node to obtain the relationship graph description.

[0140] In other embodiments of this application, the generation unit 802 is specifically used for:

[0141] For any given node, find the target node relationship from the description in Part II;

[0142] Retrieve the starting node in the target node relationship;

[0143] The starting node is determined as the dependent node of the node.

[0144] In other embodiments of this application, the generation unit 802 is specifically used for:

[0145] Determine the leaf nodes from each node;

[0146] Create a terminal node that has a dependency relationship with the leaf nodes, where the terminal node's dependency node is the leaf node;

[0147] Based on the dependent nodes of the last node, generate a first type of configuration file corresponding to the last node; and based on the node information of each node and the dependent nodes of each node, generate a second type of configuration file.

[0148] The first type of configuration file and the second type of configuration file are identified as configuration files.

[0149] In other embodiments of this application, the generation unit 802 is specifically used for:

[0150] Based on the dependent nodes of each node, determine the non-leaf nodes from each node;

[0151] Remove the non-leaf nodes from the list of all nodes and designate them as leaf nodes.

[0152] In other embodiments of this application, the generation unit 802 is specifically used for:

[0153] Find the node relationships where the target node is not empty from the description in Part 2;

[0154] Obtain the starting node from the node relationships where the target node is not empty;

[0155] Treat the starting node as a non-leaf node.

[0156] Based on the same concept, this application also provides an electronic device, such as... Figure 9 As shown, the electronic device mainly includes a processor 901, a memory 902, and a communication bus 903. The processor 901 and the memory 902 communicate with each other via the communication bus 903. The memory 902 stores programs that can be executed by the processor 901. The processor 901 executes the programs stored in the memory 902 to achieve the following steps:

[0157] From the relationship diagram editing area of ​​the visual configuration interface, obtain the relationship diagram that reflects the dependencies between tasks;

[0158] Parse the relationship graph and generate a configuration file adapted to the task scheduling system;

[0159] After the configuration file is uploaded to the task scheduling system, the scheduling information of each task can be obtained from the scheduling information configuration area of ​​the visual configuration interface. The scheduling information includes the execution time and alarm information of the project to which each task belongs.

[0160] Send scheduling information to the task scheduling system so that the task scheduling system can perform task scheduling for each task based on the configuration file and scheduling information.

[0161] The communication bus 903 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 903 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0162] The memory 902 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor 901.

[0163] The processor 901 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc., or a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0164] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the task scheduling method described above.

[0165] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape, etc.), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0166] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A task scheduling method, characterized in that, include: From the relationship diagram editing area of ​​the visual configuration interface, obtain the relationship diagram that reflects the dependencies between tasks; Parse the relationship graph to generate a configuration file adapted to the task scheduling system; After the configuration file is uploaded to the task scheduling system, the scheduling information of each task is obtained from the scheduling information configuration area of ​​the visual configuration interface. The scheduling information includes the execution time and alarm information of the project to which each task belongs. The scheduling information is sent to the task scheduling system so that the task scheduling system performs task scheduling for each task based on the configuration file and the scheduling information; The step of parsing the relationship graph and generating a configuration file adapted to the task scheduling system includes: According to a preset computer language format, a relational graph description corresponding to the relational graph is generated. The relational graph description includes a first part of the description for indicating the node information of each node and a second part of the description for indicating the node relationship of each node. Any task has a unique corresponding node in the relational graph. Obtain the node information of each node from the first part of the description, and obtain the dependent nodes of each node from the second part of the description; Based on the node information of each node and the dependent nodes of each node, a configuration file adapted to the task scheduling system is generated; The step of generating the configuration file adapted to the task scheduling system based on the node information of each node and the dependent nodes of each node includes: Determine the leaf nodes from each of the nodes; Create a terminal node that has a dependency relationship with the leaf node, wherein the dependent node of the terminal node is the leaf node; Obtain the node information of the last node, and generate a configuration file corresponding to the last node based on the node information of the last node and the dependent nodes of the last node; Based on the node information of each node and the dependent nodes of each node, generate a configuration file corresponding to each node. The configuration file corresponding to the last node and the configuration file corresponding to each node are determined as the configuration file.

2. The method according to claim 1, characterized in that, Generate a relational diagram description corresponding to the relational diagram according to a preset computer language format, including: Identify at least one node in the relationship graph according to the pre-set node attributes; Obtain the node information of each node respectively; Identify the directed line segments that are connected to each of the nodes; Based on the direction of the directed line segment, the dependent nodes of each node are determined respectively; The relationship graph description is obtained by describing the node information of each node and the dependent nodes of each node according to the preset computer language format.

3. The method according to claim 1, characterized in that, The dependent nodes of each node are obtained from the description in the second part, including: For any given node, find the target node relationship of the node described in the second part. Obtain the starting node from the target node relationship; The starting node is determined as the dependent node of the node.

4. The method according to claim 1, characterized in that, Determining leaf nodes from the aforementioned nodes includes: Based on the dependent nodes of each node, determine the non-leaf nodes from each node; The nodes remaining after removing the non-leaf nodes from the nodes are determined as the leaf nodes.

5. The method according to claim 4, characterized in that, Based on the dependent nodes of each node, non-leaf nodes are determined from the nodes, including: Find the node relationships where the target node is not empty from the description in Part II; Obtain the starting node from the node relationships where the target node is not empty; The starting node is designated as the non-leaf node.

6. A task scheduling device, characterized in that, include: The first acquisition unit is used to acquire a relationship diagram reflecting the dependencies between tasks from the relationship diagram editing area of ​​the visual configuration interface; The generation unit is used to parse the relationship graph and generate a configuration file adapted to the task scheduling system; The second acquisition unit is used to acquire the scheduling information of each task from the scheduling information configuration area of ​​the visual configuration interface after the configuration file is uploaded to the task scheduling system. The scheduling information includes the execution time and alarm information of the project to which each task belongs. A scheduling unit is configured to send the scheduling information to the task scheduling system, so that the task scheduling system performs task scheduling for each task based on the configuration file and the scheduling information. Specifically, the generation unit is used for: According to the preset computer language format, a relational graph description corresponding to the relational graph is generated. The relational graph description includes a first part of the description for indicating the node information of each node and a second part of the description for indicating the node relationship of each node. Any task has a unique corresponding node in the relational graph. Obtain the node information of each node from the first part of the description, and obtain the dependent nodes of each node from the second part of the description; Based on the node information of each node and the dependent nodes of each node, a configuration file adapted to the task scheduling system is generated. Specifically, the generation unit is used for: Determine the leaf nodes from each node; Create a terminal node that has a dependency relationship with the leaf nodes, where the terminal node's dependency node is the leaf node; Based on the dependent nodes of the last node, generate a first type of configuration file corresponding to the last node; and based on the node information of each node and the dependent nodes of each node, generate a second type of configuration file. The first type of configuration file and the second type of configuration file are identified as configuration files.

7. An electronic device, characterized in that, include: The processor, memory, and communication bus are used to communicate with each other. The memory is used to store computer programs; The processor is used to execute the program stored in the memory to implement the task scheduling method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the task scheduling method according to any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the task scheduling method according to any one of claims 1-5.

Images