DAG structure-based cluster monitoring task scheduling method and apparatus, and electronic device

Abstract

The invention discloses a DAG (directed acyclic graph) structure-based cluster monitoring task scheduling method, which comprises the following steps of: forming tasks in a cluster monitoring process into a task queue, constructing a directed acyclic graph according to a dependency relationship of each task so as to accurately represent a complex dependency relationship and an execution sequence among the tasks, and determining a scheduling sequence of each task based on the directed acyclic graph, according to the method, global optimization of task scheduling is realized, the utilization rate of computing resources is improved, so that the processing speed of large-scale data is accelerated, the tasks which can be executed in parallel are executed at the same time by utilizing the parallel computing capability of the cluster based on the directed acyclic graph and the scheduling sequence of each task, and the data processing efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to task scheduling technology, and in particular to a cluster monitoring task scheduling method, device and electronic equipment based on a DAG structure. Background Art

[0002] Cluster monitoring refers to the monitoring and management of a group of mutually collaborating computer systems to ensure that they can run efficiently and stably. It usually includes the monitoring of hardware resources (such as CPU, memory, disk space, etc.) and software services (such as databases, Web servers, etc.). This monitoring process involves the execution and scheduling of different data processing tasks.

[0003] In traditional cluster monitoring and data processing architectures, task scheduling mechanisms often use basic polling scheduling or priority queue scheduling algorithms, which are particularly inadequate when faced with complex task dependency graphs and massive data sets in modern data centers. The complexity of the task dependency graph is reflected in the intricate predecessor and successor relationships between tasks, and the existing task scheduling mechanism is difficult to efficiently parse and optimize these relationships, resulting in tasks being in a waiting state for a long time. At the same time, cluster resources are not fully utilized, resulting in significant resource idleness.

[0004] In addition, with the continuous emergence of data torrents, real-time data processing has become the cornerstone of many key applications. However, the current technical system generally lacks an efficient parallel processing framework and intelligent data scheduling strategy, which directly leads to a significant increase in the processing delay of data flowing through the system when processing PB-level or even EB-level data, making it difficult to meet the needs of low-latency sensitive applications (LLSA) such as financial transaction analysis and real-time monitoring systems. Summary of the invention

[0005] Based on this, the present invention aims to propose a cluster monitoring task scheduling method, device and electronic device based on DAG structure, introduce a directed acyclic graph to represent the cluster monitoring task dependency, and perform resource allocation and parallel execution of tasks based on this.

[0006] In a first aspect, the present invention proposes a cluster monitoring task scheduling method based on a DAG structure, comprising:

[0007] Obtain the data sources and their dependencies required for cluster monitoring;

[0008] Build a directed acyclic graph of cluster monitoring tasks based on data sources and their dependencies;

[0009] Use directed acyclic graph to determine the scheduling order of each task;

[0010] Cluster monitoring tasks are processed in parallel according to the directed acyclic graph and the scheduling order of each task.

[0011] Furthermore, a directed acyclic graph of cluster monitoring tasks is constructed based on data sources and their dependencies, including:

[0012] Determine data processing tasks based on data source dependencies;

[0013] Create nodes for each data processing task and data source, add directed edges between nodes based on the dependencies of the data sources, and generate a directed acyclic graph of the cluster monitoring task.

[0014] Furthermore, generating a directed acyclic graph of cluster monitoring tasks includes:

[0015] Generate an initial directed acyclic graph;

[0016] The initial directed acyclic graph is verified for acyclicity, and the final directed acyclic graph is generated after the verification passes.

[0017] Furthermore, using the directed acyclic graph to determine the scheduling order of each task includes:

[0018] Identify the critical task path according to the directed acyclic graph, the critical task path includes the longest path from the starting node to the end node in the directed acyclic graph;

[0019] Assign the highest priority to tasks on the critical task path, and determine the priority of other tasks outside the critical task path based on preset conditions;

[0020] Based on the priority of each task and the directed acyclic graph, the nodes that can be executed in parallel are identified, and the scheduling order of each task is determined.

[0021] Furthermore, the preset conditions include task execution time and task resource requirements.

[0022] Furthermore, the parallel processing of cluster monitoring tasks according to the directed acyclic graph and the scheduling order of each task includes:

[0023] Allocate computing resources to each task based on the current resources of the cluster and the priority of each task;

[0024] According to the directed acyclic graph and the scheduling order of each task, cluster monitoring tasks are processed in parallel using a parallel execution mechanism based on the computing resources allocated to each task.

[0025] Furthermore, data sources include system performance indicators, network traffic, and application logs.

[0026] In a second aspect, the present invention proposes a cluster monitoring task scheduling device based on a DAG structure, comprising:

[0027] Data acquisition module, used to obtain the data sources and their dependencies required for cluster monitoring;

[0028] DAG building module, which is used to build a directed acyclic graph of cluster monitoring tasks based on data sources and their dependencies;

[0029] The scheduling analysis module is used to determine the scheduling order of each task using a directed acyclic graph;

[0030] The task execution module is used to process cluster monitoring tasks in parallel according to the directed acyclic graph and the scheduling order of each task.

[0031] In a third aspect, the present invention provides an electronic device comprising a memory storing computer executable instructions and a processor, wherein when the computer executable instructions are executed by the processor, the device executes the various steps of the cluster monitoring task scheduling method based on the DAG structure provided in the first aspect.

[0032] In a fourth aspect, the present invention provides a readable storage medium storing a computer executable program, which, when executed, can implement the various steps of the cluster monitoring task scheduling method based on the DAG structure provided in the first aspect.

[0033] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0034] The present invention provides a cluster monitoring task scheduling method based on a DAG structure, which forms tasks in a cluster monitoring process into a task queue, constructs a directed acyclic graph according to the dependency relationship of each task, thereby accurately representing the complex dependency relationship and execution order between tasks, determines the scheduling order of each task based on the directed acyclic graph, realizes global optimization of task scheduling, improves the utilization rate of computing resources, thereby speeding up the processing speed of large-scale data, and utilizes the parallel computing capability of the cluster based on the directed acyclic graph and the scheduling order of each task to simultaneously execute tasks that can be executed in parallel, thereby significantly improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0036] Figure 1 This is a flow chart of a cluster monitoring task scheduling method based on a DAG structure provided by an embodiment of the present invention;

[0037] Figure 2It is a cluster monitoring multi-task dependency diagram provided by an embodiment of the present invention;

[0038] Figure 3 is based on Figure 2 Constructed directed acyclic graph;

[0039] Figure 4 Yes Figure 2 Task execution sequence diagram after parallel scheduling of multiple tasks;

[0040] Figure 5 It is a schematic diagram of the structure of a cluster monitoring task scheduling device based on a DAG structure provided by an embodiment of the present invention;

[0041] Figure 6 This is a diagram of the electronic device architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] See also Figure 1 An embodiment of the present invention provides a cluster monitoring task scheduling method based on a DAG structure, comprising the following steps:

[0044] Step S110: Obtain data sources and their dependencies required for cluster monitoring.

[0045] In this step, the data source obtained includes system-level data for monitoring the operating status and resource utilization of nodes in the cluster, including system performance indicators such as CPU usage, memory usage, disk I / O, network traffic, process monitoring, etc. The dependencies of these data can be obtained through the operating system or network interface; the data source can also include application-level data for monitoring the status of application services and tasks running in the cluster, and cluster-level data that reflects the operating status and performance of the entire cluster. In a more preferred embodiment, anomaly detection and event data can also be obtained. Analyze the dependencies between the data sources, that is, which data acquisition or processing depends on the results of other data. This dependency will be used to construct a DAG graph.

[0046] Step S120: Construct a directed acyclic graph of cluster monitoring tasks according to data sources and their dependencies.

[0047] Specifically, step S120 includes the following steps:

[0048] Determine data processing tasks based on data source dependencies;

[0049] Create nodes for each data processing task and data source, add directed edges between nodes based on the dependencies of the data sources, and generate a directed acyclic graph of the cluster monitoring task.

[0050] A node in a directed acyclic graph represents a data processing task (such as conversion, aggregation, filtering, etc.) or an action to obtain a data source. The boundary represents the dependency between nodes, that is, the data flow and processing order. Node attributes include task name, task type, data format, etc. The process of building a directed acyclic graph requires ensuring that there are no loops in the graph. If there are loops, it means that some tasks can never start executing because they depend on their own output or the output of subsequent tasks.

[0051] When constructing a directed acyclic graph, collect detailed information about all data processing tasks and data sources, including their names, types, dependencies, etc. Create a node for each data processing task and data source based on the collected information, add directed edges between nodes based on the dependencies, and ensure that the direction of the edges is correct to reflect the flow of data and the execution order of tasks. It is preferred to verify the acyclicity of the graph before outputting the final graph, during or after adding edges. The verification process can be implemented by depth-first search (DFS), topological sorting, or other graph algorithms.

[0052] In a further embodiment, the directed acyclic graph may be optimized as needed, such as merging similar tasks, reducing redundant dependencies, etc., and a visual graph representation may be generated for easier understanding and analysis.

[0053] Step S130: Determine the scheduling order of each task using a directed acyclic graph.

[0054] Specifically, step S130 includes the following steps:

[0055] Step S131. Identify a critical task path according to the directed acyclic graph, where the critical task path includes the longest path from the start node to the end node in the directed acyclic graph.

[0056] In this step, the critical path is the longest path from the starting node to the ending node in the DAG graph, which determines the shortest completion time of the entire project. Specifically, starting from the starting node of the directed acyclic graph, the earliest start time and the earliest completion time of each node are calculated step by step along the direction of the edge. If the earliest completion time of a node is equal to the earliest start time of its subsequent node plus the weight of the edge (i.e., the execution time of the task), then the edge is part of the critical task path, and such edges are gradually identified in a cycle to construct a complete critical task path. More preferably, after identifying the critical task path, it can be optimized and adjusted according to actual conditions. For example, the length of the critical task path can be shortened by increasing resources, optimizing task execution logic, or adjusting task priorities, thereby shortening the completion time of the entire project.

[0057] Step S132: Assign the highest priority to the tasks on the critical task path, and determine the priorities of other tasks outside the critical task path according to preset conditions.

[0058] Step S133: Based on the priority of each task and the directed acyclic graph, nodes that can be executed in parallel are identified to determine the scheduling order of each task.

[0059] Since the critical task path determines the shortest completion time of the entire project, tasks on the critical path will be given a higher priority. In addition to the tasks on the critical task path, the priority of other tasks can be further adjusted based on other factors, such as task execution time, task resource requirements, etc. Then, according to the current resource status of the cluster, such as CPU, memory, network bandwidth, etc., appropriate computing resources are allocated to each task. When allocating resources, the priority and resource requirements of each task need to be considered to ensure that tasks with higher priority obtain sufficient computing resources. Then, the directed acyclic graph is topologically sorted according to the priority and parallel logic to determine the scheduling order of each task. The topological sorting algorithm selects a task without a predecessor node as the starting node, and then gradually selects tasks without predecessor nodes for sorting until all nodes are sorted. For each task, calculate its earliest start time and earliest finish time. The earliest start time refers to the earliest time point at which the task can start execution, which depends on the task's predecessor tasks and the resource status of the cluster; the earliest finish time refers to the earliest time point at which the task can be completed, which is equal to the earliest start time plus the task's execution time. According to the earliest start time and earliest finish time of the task, the scheduling order of each task and the set of tasks that can be executed in parallel can be determined.

[0060] Step S140: Process cluster monitoring tasks in parallel according to the directed acyclic graph and the scheduling order of each task.

[0061] Specifically, at any time, all nodes with in-degree 0 are regarded as task groups that can be executed in parallel. The tasks in each task group are independent of each other. According to the directed acyclic graph and the scheduling order, the computing resources allocated based on the previous steps can be allocated to different computing nodes for parallel processing.

[0062] In a more preferred embodiment, computing resources can be dynamically allocated according to the resource requirements of the task and the load of the cluster; or, large tasks can be split into smaller subtasks to increase parallelism, while small tasks with simple dependencies can be merged into one large task to reduce scheduling overhead.

[0063] For example, Figures 2~4 The following diagram illustrates the situation of building a directed acyclic graph of tasks for cluster monitoring process and performing scheduling optimization, where Figure 2 This is a schematic diagram of the interdependence of multiple tasks in the cluster monitoring process. Figure 3 Yes Figure 2 A directed acyclic graph constructed by multi-task, Figure 4 is based on Figure 3 right Figure 2 Schematic diagram of the task execution order after multi-task parallel scheduling optimization.

[0064] The above embodiments provide a cluster monitoring task scheduling method based on a DAG structure, which forms tasks in the cluster monitoring process into a task queue, constructs a directed acyclic graph according to the dependency relationship of each task, thereby accurately representing the complex dependency relationship and execution order between tasks, and determines the scheduling order of each task based on the directed acyclic graph, thereby achieving global optimization of task scheduling, improving the utilization rate of computing resources, and thus speeding up the processing speed of large-scale data. Based on the directed acyclic graph and the scheduling order of each task, the parallel computing capability of the cluster is utilized to simultaneously execute tasks that can be executed in parallel, thereby significantly improving data processing efficiency.

[0065] The above disclosed method can be implemented by using various forms of equipment, so the present invention also discloses a cluster monitoring task scheduling device based on a DAG structure corresponding to the above method, and a specific embodiment is given below for detailed description.

[0066] See also Figure 5 An embodiment of the present invention provides a cluster monitoring task scheduling device based on a DAG structure, comprising:

[0067] Data acquisition module 502, used to obtain data sources and dependencies required for cluster monitoring;

[0068] DAG construction module 504, used to construct a directed acyclic graph of cluster monitoring tasks according to data sources and their dependencies;

[0069] The scheduling analysis module 506 is used to determine the scheduling order of each task using a directed acyclic graph;

[0070] The task execution module 508 is used to process cluster monitoring tasks in parallel according to the directed acyclic graph and the scheduling order of each task.

[0071] The implementation principle and technical effects of the device provided in the embodiments of the present application are the same as those of the aforementioned method embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiments.

[0072] The methods and related devices mentioned in the above embodiments are described with reference to the method flow charts and / or structural diagrams provided in the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0073] The following embodiments are described by taking the method applied to a computer device as an example. It can be understood that the computer device can be any device with computing and processing functions, and can be but not limited to a server or a personal laptop computer, etc. In one embodiment, the computer device can be an application server, and the application server can be a server for running an application to be tested.

[0074] See also Figure 6, which shows a hardware block diagram of an electronic device, the electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0075] like Figure 6 As shown, the electronic device includes: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0076] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0077] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0078] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0079] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to implement various processing flows of the aforementioned cluster monitoring task scheduling based on the DAG structure.

[0080] An embodiment of the present invention also provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the various processing flows of the cluster monitoring task scheduling solution based on the DAG structure provided in the above embodiment and / or any possible implementation method in combination with the embodiment are implemented.

[0081] The above-mentioned embodiments have described the present invention in particular detail with respect to possible scenarios, and those skilled in the art will recognize that the present invention can be practiced through other embodiments. The specific naming of components, the capitalization of terms, attributes, data structures, or any other programming or structural aspects are not mandatory or important, and the mechanisms or features of the present invention may have different names, forms, or procedures. The system may be implemented by a combination of hardware and software (as described), entirely by hardware elements, or entirely by software elements. The specific division of functions between the various system components described herein is exemplary only and not mandatory; on the contrary, the functions performed by a single system component may be performed by multiple components, or the functions performed by multiple components may be performed by a single component.

[0082] Those skilled in the art should understand that the various steps of the above disclosed method can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0083] These computing device executable programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors, and these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0084] Certain aspects of the present invention include process steps and instructions described herein in the form of algorithms. It should be noted that the process steps and instructions of the present invention can be implemented in software, firmware and / or hardware, and when implemented by software, it can be downloaded, stored on different platforms used by various operating systems and operated from the platforms.

[0085] Those skilled in the art will understand that the structures shown in the accompanying drawings are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

[0086] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "possible design" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0087] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0088] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cluster monitoring task scheduling method based on DAG structure, characterized in that: include: Obtain the data sources and their dependencies required for cluster monitoring; Constructing a directed acyclic graph of cluster monitoring tasks based on the data sources and their dependencies; Determine the scheduling order of each task using the directed acyclic graph; The cluster monitoring tasks are processed in parallel according to the directed acyclic graph and the scheduling order of each task.

2. The method according to claim 1, characterized in that The directed acyclic graph of cluster monitoring tasks constructed according to the data sources and their dependencies includes: Determine data processing tasks based on data source dependencies; Create nodes for each data processing task and data source, add directed edges between nodes based on the dependencies of the data sources, and generate a directed acyclic graph of the cluster monitoring task.

3. The method according to claim 2, characterized in that The directed acyclic graph for generating cluster monitoring tasks includes: Generate an initial directed acyclic graph; The initial directed acyclic graph is verified for acyclicity, and the final directed acyclic graph is generated after the verification passes.

4. The method according to claim 1, characterized in that: Determining the scheduling order of each task by using the directed acyclic graph includes: Identify the critical task path according to the directed acyclic graph, the critical task path includes the longest path from the starting node to the end node in the directed acyclic graph; Assign the highest priority to tasks on the critical task path, and determine the priority of other tasks outside the critical task path based on preset conditions; Based on the priority of each task and the directed acyclic graph, the nodes that can be executed in parallel are identified, and the scheduling order of each task is determined.

5. The method according to claim 4, characterized in that The preset conditions include the execution time of the task and the task resource requirements.

6. The method according to claim 1, characterized in that The parallel processing of cluster monitoring tasks according to the directed acyclic graph and the scheduling order of each task includes: Allocate computing resources to each task based on the current resources of the cluster and the priority of each task; According to the directed acyclic graph and the scheduling order of each task, cluster monitoring tasks are processed in parallel using a parallel execution mechanism based on the computing resources allocated to each task.

7. The method according to claim 1, characterized in that The data sources include system performance indicators, network traffic, and application logs.

8. A cluster monitoring task scheduling device based on DAG structure, characterized in that: include: Data acquisition module, used to obtain the data sources and their dependencies required for cluster monitoring; DAG building module, which is used to build a directed acyclic graph of cluster monitoring tasks based on data sources and their dependencies; The scheduling analysis module is used to determine the scheduling order of each task using a directed acyclic graph; The task execution module is used to process cluster monitoring tasks in parallel according to the directed acyclic graph and the scheduling order of each task.

9. An electronic device, characterized in that: The device comprises a memory storing computer executable instructions and a processor. When the computer executable instructions are executed by the processor, the device executes the cluster monitoring task scheduling method based on the DAG structure as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: A computer executable program is stored, and when the program is executed, the cluster monitoring task scheduling method based on the DAG structure as described in any one of claims 1 to 7 can be implemented.

Images