Distributed task scheduling method and device, distributed task processing system and medium

Abstract

The invention relates to a distributed task scheduling method, distributed task scheduling equipment, a distributed task processing system and a storage medium. The distributed task scheduling method comprises the following steps: determining load data of each computing node according to node resource use information of each computing node; according to the load data of each computing node, determining a first computing node of which the load exceeds a set first load amount and a plurality of second computing nodes of which the load is smaller than a set second load amount; obtaining a large task in the first computing node; splitting the large task into a plurality of sub-tasks according to a set slicing condition; and according to the task attribute of each sub-task and the node attribute and load data of each second computing node, distributing each sub-task to each second computing node, so that each second computing node executes the corresponding sub-task. According to the method, the node resources of the computing nodes in the distributed environment can be dynamically and fully utilized, and the expansibility and performance of the distributed task scheduling system are improved.

Description

Technical Field

[0001] The present application relates to the field of task processing technology, and in particular to a distributed task scheduling method and distributed task scheduling device, a distributed task processing system and a storage medium. Background Art

[0002] In modern computing environments, distributed task scheduling devices play a vital role in processing a large number of concurrent tasks. These devices are usually used to schedule and manage distributed computing resources to perform complex computing tasks. However, in high-concurrency scenarios, existing distributed task scheduling devices cannot fully utilize the computing resources in a distributed environment during task scheduling and execution, and are prone to performance bottlenecks. Summary of the invention

[0003] Based on this, it is necessary to provide a distributed task scheduling method and a distributed task scheduling device, a distributed task processing system and a storage medium to address the above technical problems, which can split large tasks in overloaded computing nodes based on the node resource usage of each computing node and distribute them to lightly loaded computing nodes for execution, dynamically and fully utilize the node resources of each computing node in a distributed environment, and improve the scalability and performance of the distributed task scheduling system.

[0004] A distributed task scheduling method, applied to a distributed task scheduling device, includes: obtaining node resource usage information of each computing node, determining the load data of each computing node according to the node resource usage information of each computing node; determining a first computing node whose load exceeds a set first load and multiple second computing nodes whose load is less than a set second load according to the load data of each computing node, wherein the first load is greater than the second load; obtaining a large task in the first computing node, wherein the large task is a task that satisfies any one or more of the following conditions: the node resources occupied by the first computing node are greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold; splitting the large task into multiple subtasks according to the set slicing conditions; obtaining the task attributes of each subtask, the node attributes and load data of each second computing node, and allocating each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, so that each second computing node executes the corresponding subtask.

[0005] In one of the embodiments, the node resource usage information includes CPU utilization, memory utilization and network bandwidth utilization; the load data of each computing node is determined based on the node resource usage information of each computing node, including: obtaining the weight coefficients of the CPU utilization, memory utilization and network bandwidth utilization of each computing node respectively; calculating the load data of each computing node based on the CPU utilization of each computing node and the weight coefficient of the CPU utilization, the memory utilization and the weight coefficient of the memory utilization, and the network bandwidth utilization and the weight coefficient of the network bandwidth utilization.

[0006] In one of the embodiments, setting slicing conditions includes data slicing conditions, task decomposition conditions and time window conditions; splitting a large task into multiple subtasks according to the set slicing conditions, including: when the task structure of the large task meets the task decomposition conditions, splitting the large task into subtasks of each task logic based on the task decomposition conditions; when the large task is a data file task, dividing the data set of the large task into multiple sub-data sets based on the row or column logic in the data slicing conditions; when the large task involves time series data, splitting the large task into multiple subtasks that meet the time series according to the time window conditions.

[0007] In one of the embodiments, the node attributes of each second computing node include CPU performance and disk read / write performance; each subtask is assigned to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, including: when the task attribute of any subtask is compute-intensive, based on the node attributes of each second computing node, a second computing node whose CPU performance meets the set CPU performance requirement is determined, and any subtask is assigned to the second computing node whose CPU performance meets the set performance requirement; when the task attribute of any subtask is data-intensive, based on the node attributes of each second computing node, a second computing node whose disk read / write performance meets the set disk read / write performance requirement is determined, and any subtask is assigned to the second computing node whose disk read / write performance meets the set disk read / write performance requirement; when the task attribute of any subtask is not data-intensive When the process is data-intensive and not compute-intensive, the data attributes of any subtask and the data attributes of the data stored in each second computing node are obtained; if the data attributes of any subtask match the data attributes of the data stored in any second computing node, any subtask is assigned to the matching second computing node; if the data attributes of any subtask do not match the data attributes of the data stored in any second computing node, a second computing node whose load is less than the set load requirement is determined based on the load data of each second computing node, and any subtask is assigned to a second computing node whose load is less than the set load requirement; wherein, when the second computing node whose load is less than the set load requirement is assigned to any subtask, the node task whose task priority is less than any subtask is suspended, and the node resources of the node task are assigned to any subtask, so as to use the allocated node resources to execute any subtask.

[0008] In one of the embodiments, after the step of determining the load data of each computing node based on the node resource usage information of each computing node, it also includes: determining multiple third computing nodes whose load is less than a set third load amount based on the load data of each computing node; if multiple third target computing nodes of the multiple third computing nodes all contain tasks with task associations, then the tasks with task associations are merged according to data dependency or process order to obtain a task set, and the task set is allocated to the designated third target computing node.

[0009] In one of the embodiments, the designated third target computing node stores the same data set or data slices that need to be processed by each task in the task set.

[0010] In one of the embodiments, a distributed task scheduling method also includes: detecting whether each computing node has a fault through a heartbeat mechanism or a health check on the equipment of each computing node; if any computing node has a fault, obtaining unexecuted tasks in the faulty computing node; determining multiple fourth computing nodes whose load is less than a set fourth load based on the load data of the computing nodes that have not failed; and splitting the unexecuted tasks and distributing them to each fourth computing node respectively.

[0011] A distributed task processing system, comprising a distributed task scheduling device and a plurality of computing nodes; the distributed task scheduling device obtains node resource usage information of each computing node, and determines the load data of each computing node according to the node resource usage information of each computing node; the distributed task scheduling device determines a first computing node whose load exceeds a set first load and a plurality of second computing nodes whose load is less than a set second load according to the load data of each computing node, wherein the first load is greater than the second load; the distributed task scheduling device obtains a large task in the first computing node, wherein the large task is a task that satisfies any one or more of the following conditions: the node resource occupied by the first computing node is greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold; the distributed task scheduling device splits the large task into a plurality of subtasks according to the set slicing conditions; the distributed task scheduling device obtains the task attributes of each subtask, the node attributes and load data of each second computing node, and allocates each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node; each second computing node executes the corresponding subtask.

[0012] The present application also provides a distributed task scheduling device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned embodiments when executing the computer program.

[0013] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned embodiments of the method are implemented.

[0014] The above-mentioned distributed task scheduling method and distributed task scheduling device, distributed task processing system and storage medium obtain node resource usage information of each computing node, determine the load data of each computing node according to the node resource usage information of each computing node; determine the first computing node whose load exceeds the set first load and multiple second computing nodes whose load is less than the set second load according to the load data of each computing node; obtain the large task in the first computing node, the large task is a task that satisfies any one or more conditions of the node resources occupied by the first computing node being greater than the set node resource threshold, the task processing time being greater than the set duration, and the task data volume being greater than the data volume threshold; split the large task into multiple subtasks according to the set slicing conditions; obtain the task attributes of each subtask, the node attributes and load data of each second computing node, and allocate each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, so that each second computing node executes the corresponding subtask.

[0015] Therefore, it is possible to split a large task in any overloaded computing node into multiple subtasks and assign them to computing nodes with too little load, so as to utilize the node resources of multiple computing nodes with too little load to process the large task in the overloaded computing node, dynamically and fully utilize the node resources of each computing node in a distributed environment, avoid overloading of some computing nodes, improve the efficient utilization of resources of each computing node in the distributed task scheduling system, and improve the scalability and performance of the distributed task scheduling system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 An application environment diagram of a distributed task scheduling method in one embodiment;

[0017] Figure 2 A schematic diagram of a process of a distributed task scheduling method in an embodiment;

[0018] Figure 3 is a structural block diagram of a distributed task scheduling device 100 in one embodiment;

[0019] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0022] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0023] The distributed task scheduling method provided in this application can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the distributed task scheduling device 100 is used to execute a distributed task scheduling method of the present application. There are multiple computing nodes, such as Figure 1 As shown, the multiple computing nodes include computing node 201, computing node 202 ... computing node 203. Specifically, the distributed task scheduling device 100 obtains the node resource usage information of each computing node, determines the load data of each computing node according to the node resource usage information of each computing node; and determines the first computing node whose load exceeds the set first load amount and the multiple second computing nodes whose load is less than the set second load amount according to the load data of each computing node. The first computing node and the second computing node can be any one of computing node 201, computing node 202 ... computing node 203. The distributed task scheduling device 100 obtains a large task in the first computing node, where the large task satisfies any one or more of the following conditions: the node resources occupied by the first computing node are greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold. The large task is split into multiple subtasks according to the set slicing conditions; the task attributes of each subtask, the node attributes and load data of each second computing node are obtained, and each subtask is allocated to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, so that each second computing node executes the corresponding subtask.

[0024] In one embodiment, a distributed task scheduling method provided by the present application is applied to Figure 1 In the distributed task scheduling device 100 shown in FIG. Figure 2 As shown, a distributed task scheduling method includes the following steps:

[0025] S202, obtaining node resource usage information of each computing node, and determining load data of each computing node according to the node resource usage information of each computing node.

[0026] In this embodiment, each computing node has node resources, and uses the node resources to run the corresponding configured tasks. For example, node resources include CPU, memory, network and other resources. In actual application scenarios, each computing node runs one or more tasks, which occupy part of the node resources of the computing node. By calculating the node resource usage of the computing node, the load data of the computing node can be calculated, and the load data can reflect the current load of the computing node.

[0027] In one embodiment, the node resource usage information includes CPU utilization, memory utilization and network bandwidth utilization; the above-mentioned determining the load data of each computing node based on the node resource utilization information of each computing node includes: respectively obtaining the weight coefficients of the CPU utilization, memory utilization and network bandwidth utilization of each computing node; calculating the load data of each computing node based on the CPU utilization of each computing node and the weight coefficient of the CPU utilization, the memory utilization and the weight coefficient of the memory utilization, and the network bandwidth utilization and the weight coefficient of the network bandwidth utilization.

[0028] Specifically, each computing node is monitored and analyzed in real time. The CPU, memory, network and other resource usage of each computing node is monitored in real time. Based on the monitoring data, the load of each computing node is dynamically analyzed. The load data of each computing node is calculated as follows: the load data of the computing node = α*CUP utilization + β*memory utilization + γ*network bandwidth utilization.

[0029] Among them, α, β, and γ are the corresponding weight coefficients, reflecting the relative importance of the corresponding indicators in the comprehensive load calculation of the computing node.

[0030] S204: Determine, based on the load data of each computing node, a first computing node whose load exceeds a set first load amount and a plurality of second computing nodes whose load is less than a set second load amount.

[0031] In this embodiment, a first load and a second load are preset, and the first load is greater than the second load. When the load exceeds the first load, it is determined that the computing node is in an overloaded state, and one or more tasks need to be migrated to other computing nodes. When the load is less than the second load, it is determined that the computing node is in an underloaded state and can accept one or more tasks to be reallocated.

[0032] S206, obtaining a large task in the first computing node, where the large task is a task that satisfies any one or more of the following conditions: the node resources occupied by the first computing node are greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold.

[0033] In this embodiment, the first computing node is a computing node with an overload. Identify and obtain large tasks from the first computing node. Large tasks refer to tasks that occupy more node resources of the first computing node than the set node resource threshold when executed, that is, tasks that need to occupy too many node resources to run. Or, tasks that have a task processing time from execution to completion that is greater than the set duration, that is, tasks that run too long. Or, tasks that need to call a task data volume greater than the data volume threshold when executed. Or, tasks that meet any two of the above conditions or meet three of the conditions at the same time.

[0034] S208: Split the large task into multiple subtasks according to the set slicing conditions.

[0035] In this embodiment, a slicing condition is preset, and the large task is split into multiple subtasks according to the set slicing condition.

[0036] In one embodiment, the slicing conditions are set to include data slicing conditions, task decomposition conditions, and time window conditions; the above-mentioned splitting of a large task into multiple subtasks according to the set slicing conditions includes: when the task structure of the large task meets the task decomposition conditions, splitting the large task into subtasks of each task logic based on the task decomposition conditions; when the large task is a data file task, the data set of the large task is split into multiple sub-data sets based on the row or column logic in the data slicing conditions; when the large task involves time series data, splitting the large task into multiple subtasks that meet the time series according to the time window conditions.

[0037] Specifically, the large task is divided into several small subtasks and assigned to different computing nodes for execution.

[0038] Task slicing: Identify tasks with high node resource requirements (CPU, memory, bandwidth, or any combination thereof), long processing time, large data volume, and high task complexity as large tasks. When a task is detected as a "large task", split it into multiple smaller subtasks, usually using the following methods:

[0039] 1) Data Slicing: Split the data set of a large task into rows, columns, or other logical methods. For example, split a large data file into multiple small files by row.

[0040] 2) Task Decomposition: Decompose a complex task into multiple independent subtasks. For example, in a data analysis task, the steps of data cleaning, preprocessing, analysis, and report generation are divided into independent subtasks.

[0041] 3) Time Window: If the task involves time series data, the task can be split by time window, such as by hour, day or month.

[0042] S210, obtaining the task attributes of each subtask, the node attributes and load data of each second computing node, and assigning each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, so that each second computing node executes the corresponding subtask.

[0043] In this embodiment, each subtask has task attributes. The second computing node that undertakes each subtask also has node attributes, and each second computing node also has load data because it runs other tasks. Each subtask is assigned to a matching second computing node according to the task attributes of each subtask, the node attributes of each second computing node, and the load data. After the second computing node is assigned the corresponding subtask, it executes the assigned subtask.

[0044] In the embodiment of the present application, task allocation is dynamically adjusted according to the load of the computing nodes to avoid overloading of certain computing nodes. Based on the load evaluation results of each computing node, it is determined whether task allocation adjustment is needed. For example, if the combined load of the first computing node exceeds 80%, it is considered that the load is too heavy. If the combined load of the second computing node is less than 20%, it is considered that the load is too light.

[0045] Assumption: Compute node A and compute node B each handle multiple data processing tasks. Current monitoring data shows that the overall load of compute node A reaches 90%, while the overall load of compute node B is only 30%. Operation steps:

[0046] 1. Analyze load data: It is detected that computing node A is overloaded.

[0047] 2. Task migration decision: decide to migrate some tasks on computing node A to computing node B.

[0048] 3. Implement task migration: Select the large task to be migrated, such as the data processing task with ID T123. Save the data state of task T123 and transfer it to computing node B. Start task T123 on computing node B and restore its execution state.

[0049] Result: The load of computing node A is reduced to a reasonable level (such as 70%). The load of computing node B is increased, but still within an acceptable range (such as 50%).

[0050] Feedback and optimization: After task migration and adjustment are completed, continue to monitor the load of each computing node to ensure that the adjusted configuration can meet the system performance requirements. If new load imbalance is found, continue to make adjustments.

[0051] In one embodiment, the node attributes of each second computing node include CPU performance and disk read / write performance; the above-mentioned allocating each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node includes: when the task attribute of any subtask is compute-intensive, based on the node attributes of each second computing node, determining a second computing node whose CPU performance meets the set CPU performance requirement, and allocating any subtask to the second computing node whose CPU performance meets the set performance requirement; when the task attribute of any subtask is data-intensive, based on the node attributes of each second computing node, determining a second computing node whose disk read / write performance meets the set disk read / write performance requirement, and allocating any subtask to the second computing node whose disk read / write performance meets the set disk read / write performance requirement; when the task attribute of any subtask is not data-intensive When the process is data-intensive and not compute-intensive, the data attributes of any subtask and the data attributes of the data stored in each second computing node are obtained; if the data attributes of any subtask match the data attributes of the data stored in any second computing node, any subtask is assigned to the matching second computing node; if the data attributes of any subtask do not match the data attributes of the data stored in any second computing node, a second computing node whose load is less than the set load requirement is determined based on the load data of each second computing node, and any subtask is assigned to a second computing node whose load is less than the set load requirement; wherein, when the second computing node whose load is less than the set load requirement is assigned to any subtask, the node task whose task priority is less than any subtask is suspended, and the node resources of the node task are assigned to any subtask, so as to use the allocated node resources to execute any subtask.

[0052] Specifically, when allocating subtasks to appropriate computing nodes, the following factors are usually considered:

[0053] Compute node capabilities: Select the most suitable compute node based on the specific needs of the subtask. For example, compute-intensive subtasks are preferentially assigned to compute nodes with high-performance CPUs, and data-intensive subtasks are preferentially assigned to compute nodes with high-speed disk I / O. If neither is compute-intensive nor data-intensive, data locality is further considered. That is, the subtask is assigned to the compute node that stores the task-related data to reduce network transmission time and bandwidth consumption. If there is no compute node related to the storage task, then consider the current load of each compute node and preferentially assign the subtask to the compute node with lighter load to balance resource utilization. For example, comprehensively consider the load of indicators such as CPU, memory, and network bandwidth.

[0054] When any second computing node receives the assigned subtask, it processes the subtask and the tasks that have been executed and are to be executed in the second computing node according to the task priority. That is, node resources are allocated according to the urgency and priority of the task, and high-priority tasks can preempt the node resources of low-priority tasks.

[0055] The above-mentioned distributed task scheduling method can split a large task in any overloaded computing node into multiple subtasks and assign them to computing nodes with too little load respectively, so as to utilize the node resources of multiple computing nodes with too little load to process the large task in the overloaded computing node, dynamically and fully utilize the node resources of each computing node in a distributed environment, avoid overloading of certain computing nodes, improve the efficient utilization of resources of each computing node in the distributed task scheduling system, and improve the scalability and performance of the distributed task scheduling system.

[0056] In one embodiment, after the above step of determining the load data of each computing node according to the node resource usage information of each computing node, it also includes: determining multiple third computing nodes whose load is less than the set third load according to the load data of each computing node; if multiple third target computing nodes of the multiple third computing nodes all contain tasks associated with tasks, then merge the tasks associated with tasks according to data dependency or process order to obtain a task set, and assign the task set to the designated third target computing node. Wherein, the designated third target computing node stores the same data set or data slice that each task in the task set needs to process.

[0057] In this embodiment, the set third load amount can be equal to or less than the above-mentioned second load amount, and the third computing node can be the same as or different from the second computing node. In this embodiment, multiple tasks with task association in multiple computing nodes with too light load are merged in a data dependency or process order manner, and after obtaining the corresponding task set, the task set is assigned to the specified third target computing node. The third target computing node can be any third computing node. Therefore, the processing efficiency of multiple tasks with task association can be improved. In an example, the third target computing node can also be a computing node that specifies the same data set or data fragmentation that needs to be processed by each task in the task set. Therefore, the number of data transmissions can be reduced, data locality can be improved, and the processing efficiency of tasks can be improved.

[0058] Specifically, multiple small tasks in multiple computing nodes are merged into a large task to improve execution efficiency. The purpose of merging small tasks into a large task is usually to improve processing efficiency, reduce scheduling overhead, optimize resource utilization, or simplify processing procedures. However, not all small tasks can or should be merged into a large task. The necessity and feasibility of merging usually depends on the following factors:

[0059] 1) Task relevance: The relevance between small tasks is an important factor in determining whether to merge. If there is data dependency, process sequence, or common goals between small tasks, merging them into a large task may be more effective.

[0060] 2) Resource utilization optimization: In some cases, merging small tasks can better utilize computing resources. For example, when the small tasks themselves are relatively lightweight, frequent scheduling of these small tasks may bring unnecessary scheduling overhead. After merging, tasks can occupy more CPU time slices and reduce the overhead of context switching.

[0061] 3) Data locality: If small tasks need to process the same data set or data shards, merging these tasks can reduce the number of data transfers, improve data locality, and thus improve efficiency.

[0062] 4) Simplified task management: Merging small tasks can simplify task management and monitoring, especially when the number of tasks is very large. Managing fewer tasks can reduce the burden on the scheduling system and simplify error tracking and handling.

[0063] In one embodiment, before the above-mentioned step of obtaining the node resource usage information of each computing node, or after the step of allocating each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, a distributed task scheduling method also includes: detecting whether each computing node has a fault through a heartbeat mechanism or a health check on the equipment of each computing node; if any computing node has a fault, obtaining unexecuted tasks in the faulty computing node; determining multiple fourth computing nodes whose load is less than a set fourth load amount according to the load data of the computing nodes that have not failed; and splitting the unexecuted tasks and allocating them to each fourth computing node respectively.

[0064] In this embodiment, the health status of each computing node can be monitored and managed before executing distributed task scheduling, or after splitting a large task into multiple subtasks and assigning each subtask to a corresponding computing node for execution. Specifically, it includes:

[0065] (1) Fault detection of computing nodes: Regularly check the health status of each computing node and determine whether a computing node is faulty through a heartbeat mechanism or health check.

[0066] (2) Automatic recovery mechanism: When a computing node failure is detected, the recovery mechanism is automatically triggered to reallocate tasks on the failed computing node to other normal computing nodes for execution. The reallocation will break up all unexecuted tasks on the failed computing node, perform a calculation analysis on the comprehensive load of the existing normal computing nodes, and reallocate the tasks to normal computing nodes for execution.

[0067] (3) Task retry strategy: For tasks that fail to execute or are not completed due to network anomalies, they are automatically retried until the task is successfully completed.

[0068] (4) Data persistence and logging: The execution status and logs of tasks are persistently stored to ensure that unfinished tasks can be continued after failure recovery.

[0069] The distributed task scheduling device 100 for executing the above-mentioned distributed task scheduling method, in one embodiment, is as follows: Figure 3 As shown in Figure 1, it includes real-time monitoring and analysis module, task scheduling module, task execution module and fault-tolerant processing module. Figure 3 As shown:

[0070] The task scheduling module includes a task priority scheduling submodule and a task dependency analysis submodule. The scheduling algorithm implemented is:

[0071] (1) Priority-based scheduling algorithm:

[0072] Each task is assigned a priority, which is determined based on the importance and urgency of the task. The scheduler processes high priority tasks first, then processes low priority tasks, ensuring that important tasks are completed first.

[0073] (2) Task dependency analysis:

[0074] The system identifies the dependencies between tasks and ensures that dependent tasks can only be executed after the predecessor tasks are completed. The task dependencies are represented using a directed acyclic graph (DAG) to ensure the correct order of task execution.

[0075] The real-time monitoring and analysis module monitors the CPU, memory, network and other resource usage of each computing node in real time.

[0076] The task execution module includes the task execution submodule and the sharding and merging submodule. Specific implementation: Based on the monitoring data, dynamically analyze the load of each computing node and adjust the task allocation. Task allocation adjustment includes large task sharding and small task merging.

[0077] The fault-tolerant processing module includes a fault detection submodule and a recovery and retry submodule. The fault detection submodule regularly detects the health status of each computing node through heartbeat detection or health check. The recovery and retry submodule automatically triggers the fault recovery mechanism to reallocate tasks on the faulty node to other normal computing nodes. In addition, tasks that fail to execute or are not completed due to network anomalies are automatically retried.

[0078] The distributed task scheduling method of the above embodiment has the following technical advantages:

[0079] 1. High performance and scalability: It can run efficiently in a high-concurrency task environment and has good scalability to adapt to the continuous increase in task volume and number of nodes.

[0080] (1) The scheduling algorithm ensures that urgent and high-priority tasks are processed first and avoids task queuing.

[0081] (2) The load balancing strategy evenly distributes tasks to each computing node to avoid overloading of some computing nodes, thus ensuring efficient use of resources and system response speed.

[0082] (3) When a computing node fails, the fault-tolerant mechanism quickly detects and reallocates its tasks to other computing nodes to avoid task loss or delay.

[0083] These measures together ensure that the system can maintain efficient scheduling performance and system stability when the task volume and the number of computing nodes increase.

[0084] 2. Optimized task scheduling: Through advanced scheduling algorithms and task dependency analysis, ensure that tasks are correctly executed according to priority and dependency.

[0085] 3. Strong fault tolerance: In the event of computing node failure or network anomaly, the system can automatically recover to ensure the continuity and stability of task execution.

[0086] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0087] In one embodiment, a distributed task processing system is also provided. Figure 1As shown, a distributed task processing system includes a distributed task scheduling device 100 and multiple computing nodes. The distributed task scheduling device obtains node resource usage information of each computing node, and determines the load data of each computing node according to the node resource usage information of each computing node; the distributed task scheduling device determines a first computing node whose load exceeds a set first load and multiple second computing nodes whose load is less than a set second load according to the load data of each computing node, and the first load is greater than the second load; the distributed task scheduling device obtains a large task in the first computing node, and the large task is a task that satisfies any one or more conditions of occupying node resources of the first computing node greater than a set node resource threshold, task processing time greater than a set duration, and task data volume greater than a data volume threshold; the distributed task scheduling device splits the large task into multiple subtasks according to the set slicing conditions; the distributed task scheduling device obtains the task attributes of each subtask, the node attributes and load data of each second computing node, and allocates each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node; each second computing node executes the corresponding subtask.

[0088] For the specific definition of a distributed task processing system, please refer to the definition of a distributed task scheduling method above, which will not be repeated here.

[0089] In one embodiment, a distributed task scheduling device is provided. The distributed task scheduling device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The distributed task scheduling device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the distributed task scheduling device is used to provide computing and control capabilities. The memory of the distributed task scheduling device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the distributed task scheduling device is used to communicate with an external computing node through a network connection. When the computer program is executed by the processor, a distributed task scheduling method is implemented.

[0090] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] In one embodiment, a distributed task scheduling device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: obtaining node resource usage information of each computing node, determining load data of each computing node according to the node resource usage information of each computing node; determining a first computing node whose load exceeds a set first load and a plurality of second computing nodes whose load is less than a set second load according to the load data of each computing node, wherein the first load is greater than the second load; obtaining a large task in the first computing node, wherein the large task is a task that satisfies any one or more of the following conditions: the node resources occupied by the first computing node are greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold; splitting the large task into a plurality of subtasks according to the set slicing conditions; obtaining task attributes of each subtask, node attributes and load data of each second computing node, and allocating each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, so that each second computing node executes the corresponding subtask.

[0092] In one embodiment, the node resource usage information includes CPU utilization, memory utilization and network bandwidth utilization; when the processor executes a computer program to implement the step of determining the load data of each computing node according to the node resource usage information of each computing node, the following steps are implemented: respectively obtaining the weight coefficients of the CPU utilization, memory utilization and network bandwidth utilization of each computing node; calculating the load data of each computing node according to the CPU utilization of each computing node and the weight coefficient of the CPU utilization, the memory utilization and the weight coefficient of the memory utilization, and the network bandwidth utilization and the weight coefficient of the network bandwidth utilization.

[0093] In one embodiment, the slicing conditions are set to include data slicing conditions, task decomposition conditions and time window conditions; when the processor executes a computer program to implement the step of splitting a large task into multiple subtasks according to the set slicing conditions, the following steps are implemented: when the task structure of the large task meets the task decomposition conditions, the large task is split into subtasks of each task logic based on the task decomposition conditions; when the large task is a data file task, the data set of the large task is divided into multiple sub-data sets based on the row or column logic in the data slicing conditions; when the large task involves time series data, the large task is split into multiple subtasks that meet the time series according to the time window conditions.

[0094] In one embodiment, the node attributes of each second computing node include CPU performance and disk read / write performance; when the processor executes a computer program to implement the step of allocating each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, the following steps are implemented: when the task attribute of any subtask is computationally intensive, based on the node attributes of each second computing node, a second computing node whose CPU performance meets the set CPU performance requirement is determined, and any subtask is allocated to the second computing node whose CPU performance meets the set performance requirement; when the task attribute of any subtask is data intensive, based on the node attributes of each second computing node, a second computing node whose disk read / write performance meets the set disk read / write performance requirement is determined, and any subtask is allocated to the second computing node whose disk read / write performance meets the set disk read / write performance requirement; in any subtask When the task attributes of the service are neither data-intensive nor compute-intensive, the data attributes of any subtask and the data attributes of the data stored in each second computing node are obtained. If the data attributes of any subtask match the data attributes of the data stored in any second computing node, any subtask is assigned to the matching second computing node. If the data attributes of any subtask do not match the data attributes of the data stored in any second computing node, a second computing node with a load less than a set load requirement is determined based on the load data of each second computing node, and any subtask is assigned to a second computing node with a load less than the set load requirement. When the second computing node with a load less than the set load requirement is assigned to any subtask, the node task with a task priority less than any subtask is suspended, and the node resources of the node task are assigned to any subtask, so as to use the allocated node resources to execute any subtask.

[0095] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining multiple third computing nodes whose loads are less than a set third load amount based on the load data of each computing node; if multiple third target computing nodes of the multiple third computing nodes all contain tasks with task associations, then merging the tasks with task associations according to data dependency or process order to obtain a task set, and assigning the task set to the designated third target computing node.

[0096] In one embodiment, the designated third target computing node stores the same data set or data slice that needs to be processed by each task in the task set.

[0097] In one embodiment, when the processor executes the computer program, the following steps are also implemented: detecting whether each computing node has a fault through a heartbeat mechanism or a health check on the equipment of each computing node; if any computing node has a fault, obtaining the unexecuted tasks in the faulty computing node; determining multiple fourth computing nodes whose load is less than a set fourth load based on the load data of the computing nodes that have not failed; and splitting the unexecuted tasks and distributing them to each fourth computing node respectively.

[0098] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining node resource usage information of each computing node, and determining the load data of each computing node according to the node resource usage information of each computing node; determining a first computing node whose load exceeds a set first load and multiple second computing nodes whose load is less than a set second load according to the load data of each computing node, wherein the first load is greater than the second load; obtaining a large task in the first computing node, wherein the large task is a task that satisfies any one or more of the following conditions: the node resources occupied by the first computing node are greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold; splitting the large task into multiple subtasks according to the set slicing conditions; obtaining the task attributes of each subtask, the node attributes and load data of each second computing node, and allocating each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, so that each second computing node executes the corresponding subtask.

[0099] In one embodiment, the node resource usage information includes CPU utilization, memory utilization and network bandwidth utilization; when the computer program is executed by the processor to implement the step of determining the load data of each computing node according to the node resource usage information of each computing node, the following steps are implemented: respectively obtaining the weight coefficients of the CPU utilization, memory utilization and network bandwidth utilization of each computing node; calculating the load data of each computing node according to the CPU utilization and the weight coefficient of the CPU utilization, the memory utilization and the weight coefficient of the memory utilization, and the network bandwidth utilization and the weight coefficient of the network bandwidth utilization of each computing node.

[0100] In one embodiment, the slicing conditions are set to include data slicing conditions, task decomposition conditions and time window conditions; when the computer program is executed by the processor to implement the step of splitting a large task into multiple subtasks according to the set slicing conditions, the following steps are implemented: when the task structure of the large task meets the task decomposition conditions, the large task is split into subtasks of each task logic based on the task decomposition conditions; when the large task is a data file task, the data set of the large task is divided into multiple sub-data sets based on the row or column logic in the data slicing conditions; when the large task involves time series data, the large task is split into multiple subtasks that meet the time series according to the time window conditions.

[0101] In one embodiment, the node attributes of each second computing node include CPU performance and disk read / write performance; when the computer program is executed by the processor to implement the step of allocating each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, the following steps are implemented: when the task attribute of any subtask is compute-intensive, based on the node attributes of each second computing node, a second computing node whose CPU performance meets the set CPU performance requirements is determined, and any subtask is allocated to the second computing node whose CPU performance meets the set performance requirements; when the task attribute of any subtask is data-intensive, based on the node attributes of each second computing node, a second computing node whose disk read / write performance meets the set disk read / write performance requirements is determined, and any subtask is allocated to the second computing node whose disk read / write performance meets the set disk read / write performance requirements; in any subtask When the task attributes of the service are neither data-intensive nor compute-intensive, the data attributes of any subtask and the data attributes of the data stored in each second computing node are obtained. If the data attributes of any subtask match the data attributes of the data stored in any second computing node, any subtask is assigned to the matching second computing node. If the data attributes of any subtask do not match the data attributes of the data stored in any second computing node, a second computing node with a load less than a set load requirement is determined based on the load data of each second computing node, and any subtask is assigned to a second computing node with a load less than the set load requirement. When the second computing node with a load less than the set load requirement is assigned to any subtask, the node task with a task priority less than any subtask is suspended, and the node resources of the node task are assigned to any subtask, so as to use the allocated node resources to execute any subtask.

[0102] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining multiple third computing nodes whose loads are less than a set third load based on the load data of each computing node; if multiple third target computing nodes of the multiple third computing nodes all contain tasks with task associations, then merging the tasks with task associations according to data dependency or process order to obtain a task set, and assigning the task set to the designated third target computing node.

[0103] In one embodiment, the designated third target computing node stores the same data set or data slice that needs to be processed by each task in the task set.

[0104] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: detecting whether each computing node has a fault through a heartbeat mechanism or a health check on the equipment of each computing node; if any computing node has a fault, obtaining the unexecuted tasks in the faulty computing node; determining multiple fourth computing nodes whose load is less than a set fourth load based on the load data of the computing nodes that have not failed; and splitting the unexecuted tasks and distributing them to each fourth computing node respectively.

[0105] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0106] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A distributed task scheduling method, applied to a distributed task scheduling device, the method comprising: Obtaining node resource usage information of each computing node, and determining load data of each computing node according to the node resource usage information of each computing node; Determine, according to the load data of each computing node, a first computing node whose load exceeds a set first load amount and a plurality of second computing nodes whose load is less than a set second load amount, wherein the first load amount is greater than the second load amount; Obtain a large task in the first computing node, where the large task is a task that satisfies any one or more of the following conditions: the node resources occupied by the first computing node are greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold; Splitting the large task into multiple subtasks according to the set slicing conditions; Obtain the task attributes of each subtask, the node attributes and load data of each second computing node, and assign each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node, so that each second computing node executes the corresponding subtask.

2. The method according to claim 1, characterized in that The node resource usage information includes CPU utilization, memory utilization and network bandwidth utilization; Determining the load data of each computing node according to the node resource usage information of each computing node includes: Obtain the weight coefficients of the CPU utilization, memory utilization, and network bandwidth utilization of each computing node respectively; The load data of each computing node is calculated according to the CPU utilization and the weight coefficient of the CPU utilization, the memory utilization and the weight coefficient of the memory utilization, and the network bandwidth utilization and the weight coefficient of the network bandwidth utilization of each computing node.

3. The method according to claim 1, characterized in that: The set slicing conditions include data slicing conditions, task decomposition conditions and time window conditions; The step of splitting the large task into multiple subtasks according to the set slicing conditions includes: When the task structure of the large task meets the task decomposition condition, splitting the large task into subtasks of each task logic based on the task decomposition condition; When the large task is a data file task, the data set of the large task is divided into a plurality of sub-data sets based on the row or column logic in the data slicing condition; When the large task involves time series data, the large task is split into a plurality of subtasks satisfying the time series according to the time window condition.

4. The method according to claim 1, characterized in that: The node attributes of each second computing node include CPU performance and disk read and write performance; Allocating each subtask to each second computing node according to the task attribute of each subtask and the node attribute and load data of each second computing node includes: When the task attribute of any subtask is computationally intensive, a second computing node whose CPU performance meets the set CPU performance requirement is determined based on the node attributes of each second computing node, and any subtask is allocated to the second computing node whose CPU performance meets the set performance requirement; When the task attribute of any subtask is data intensive, determine the second computing node whose disk read / write performance meets the set disk read / write performance requirement based on the node attribute of each second computing node, and assign the any subtask to the second computing node whose disk read / write performance meets the set disk read / write performance requirement; When the task attributes of any subtask are neither data-intensive nor compute-intensive, the data attributes of any subtask and the data attributes of the data stored in each second computing node are obtained; if the data attributes of any subtask match the data attributes of the data stored in any second computing node, the subtask is assigned to the matching second computing node; if the data attributes of any subtask do not match the data attributes of the data stored in any second computing node, a second computing node whose load is less than a set load requirement is determined based on the load data of each second computing node, and the subtask is assigned to the second computing node whose load is less than the set load requirement; wherein, when the second computing node whose load is less than the set load requirement is assigned to the subtask, the node task whose task priority is less than the subtask is suspended, and the node resources of the node task are assigned to the subtask, so as to use the allocated node resources to execute the subtask.

5. The method according to claim 4, characterized in that After the step of determining the load data of each computing node according to the node resource usage information of each computing node, the method further includes: Determine, according to the load data of each computing node, a plurality of third computing nodes whose loads are less than a set third load amount; If multiple third target computing nodes of multiple third computing nodes all contain task-related tasks, then the task-related tasks are merged according to data dependency or process order to obtain a task set, and the task set is allocated to the designated third target computing node.

6. The method according to claim 5, characterized in that The designated third target computing node stores the same data set or data slices that need to be processed by each task in the task set.

7. The method according to claim 4, characterized in that The method further comprises: Detect whether each computing node has a fault by using a heartbeat mechanism or performing a health check on the equipment of each computing node; If any computing node fails, the tasks not executed in the failed computing node are obtained; Determine, based on the load data of the computing nodes that have not failed, a plurality of fourth computing nodes whose loads are less than a set fourth load amount; The unexecuted tasks are split and respectively allocated to the fourth computing nodes.

8. A distributed task processing system, comprising a distributed task scheduling device and a plurality of computing nodes; The distributed task scheduling device obtains node resource usage information of each computing node, and determines load data of each computing node according to the node resource usage information of each computing node; The distributed task scheduling device determines, according to the load data of each computing node, a first computing node whose load exceeds a set first load amount and a plurality of second computing nodes whose load is less than a set second load amount, wherein the first load amount is greater than the second load amount; The distributed task scheduling device obtains a large task in the first computing node, where the large task satisfies any one or more of the following conditions: the node resources occupied by the first computing node are greater than a set node resource threshold, the task processing time is greater than a set duration, and the task data volume is greater than a data volume threshold; The distributed task scheduling device splits the large task into multiple subtasks according to the set slicing conditions; The distributed task scheduling device obtains the task attributes of each subtask, the node attributes and load data of each second computing node, and allocates each subtask to each second computing node according to the task attributes of each subtask and the node attributes and load data of each second computing node; Each second computing node executes a corresponding subtask.

9. A distributed task scheduling device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images