Task scheduling method and device, electronic equipment and storage medium

Abstract

The invention provides a task scheduling method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed calculation task cached by a scheduling node, broadcasting a first calculation task to a plurality of calculation nodes, carrying out the calculation according to the duration of processing the first calculation task by each calculation node, and obtaining a calculation power score of each calculation node, and determining a target computing node of the to-be-allocated computing task according to the computing power score of the computing node, and scheduling the second computing task to the target computing node according to the computing power score. According to the embodiment of the invention, the computing power capabilities of the computing nodes are uniformly evaluated, task allocation is timely intervened according to the computing power capabilities of different computing nodes, and the computing tasks are scheduled to the computing nodes meeting the computing power, so that effective task scheduling of various types of computing nodes is realized, and the task scheduling efficiency is improved. And the effective utilization rate of the computing power resources of the computing nodes in the heterogeneous cluster is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a task scheduling method, device, electronic equipment and storage medium. Background Art

[0002] In order to cope with the huge computing load brought by massive data processing, the field of high-performance computing is undergoing a transformation - the heterogeneity of computing resources. Heterogeneous computing refers to the integration of different types of computing units (such as CPU, GPU, NPU, etc.) on the same system or platform to achieve higher computing efficiency and lower energy consumption. With the complexity of application scenarios, model reasoning tasks have become more and more time-consuming. Especially after the rise of large models, the capabilities of a single reasoning node are no longer sufficient to meet the needs of complex business scenarios. Therefore, reasoning node clusters have become an inevitable choice. Introducing different types of computing nodes in reasoning node clusters can not only make more effective use of existing computing resources, but also break the dependence on a single type or platform.

[0003] Existing cluster scheduling implementation methods usually perform task scheduling for clusters composed of computing units of the same type. When allocating tasks, only the cluster load needs to be considered, without considering the capabilities of different nodes. It is difficult to take into account the processing capabilities of various types of nodes, resulting in the inability to meet the task allocation needs of heterogeneous clusters. Summary of the invention

[0004] In view of this, the present invention aims to propose a task scheduling method, device, electronic device and storage medium to solve the problem that different nodes are not considered when allocating tasks, it is difficult to take into account the processing capabilities of other types of nodes, and the task allocation requirements of heterogeneous clusters cannot be met.

[0005] According to a first aspect of the present invention, a task scheduling method is provided, which is applied to a scheduling cluster, wherein the scheduling cluster includes a scheduling node and a plurality of computing nodes, and the scheduling node communicates with the plurality of computing nodes. The method includes:

[0006] Obtaining the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task;

[0007] Broadcasting the first computing task to the plurality of computing nodes, and calculating a computing power score of each computing node according to a time duration for each computing node to process the first computing task;

[0008] Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node;

[0009] The second computing task is dispatched to the target computing node according to the computing power score.

[0010] Optionally, broadcasting the first computing task to the plurality of computing nodes, and calculating a computing power score of each computing node according to a time duration for each computing node to process the first computing task, includes:

[0011] In response to a computing power evaluation request for the computing node, broadcasting the first computing task to a plurality of the computing nodes;

[0012] Obtaining a maximum completion time and an average completion time of the computing node processing the first computing task;

[0013] The maximum completion time and the average completion time are used to calculate the computing power score of each computing node.

[0014] Optionally, the calculating the computing power score of each computing node by using the maximum completion time and the average completion time includes:

[0015] The maximum completion time and the average completion time are normalized, and a comprehensive score of the computing node is obtained according to the normalized maximum completion time and the average completion time;

[0016] The comprehensive scores of the computing nodes are rounded and converted to integers to obtain the computing power scores of the computing nodes.

[0017] Optionally, the scheduling the second computing task to the target computing node according to the computing power score includes:

[0018] Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node;

[0019] Determine the number of tasks to be assigned to the target computing node according to the computing power score of the target computing node;

[0020] Storing the second computing task in a preset task scheduling slot list according to the task quantity of the target computing node;

[0021] The second computing task in the preset task scheduling slot list is scheduled to the target computing node.

[0022] Optionally, after determining the target computing node to be assigned the computing task according to the computing power score of the computing node, the method further includes:

[0023] Monitor and update the computing power scores of the computing nodes, and determine computing nodes whose computing power scores are lower than a preset computing power threshold;

[0024] The computing nodes whose computing power scores are lower than the preset computing power threshold are removed from the target computing nodes to be assigned computing tasks.

[0025] Optionally, after determining the target computing node to be assigned the computing task according to the computing power score of the computing node, the method further includes:

[0026] Determine the available computing power of the target computing node, judge whether the available computing power of the target computing node meets the number of tasks of the target computing node, and determine the abnormal computing node;

[0027] Monitor whether the available computing power of the abnormal computing node is restored within a preset resource-restricted time period.

[0028] If the available computing power of the abnormal computing node is not restored within the preset resource restriction time, the target computing node to which the computing task is to be assigned is re-determined.

[0029] Optionally, after determining the target computing node to be assigned the computing task according to the computing power score of the computing node, and scheduling the second computing task to the target computing node according to the computing power score, the method further includes:

[0030] Monitoring the remaining computing power of the target computing node;

[0031] Determining whether the target computing node exceeds the limit according to the preset computing power resource upper limit of the target computing node and the remaining computing power;

[0032] If so, the excess computing node is reported to the scheduling node, and the allocation of computing tasks to the excess computing node is stopped.

[0033] According to a second aspect of the present invention, there is provided a task scheduling device, which is applied to a scheduling cluster, wherein the scheduling cluster includes a scheduling node and a plurality of computing nodes, and the scheduling node communicates with the plurality of computing nodes, and the device includes:

[0034] An acquisition task module, used to acquire the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task;

[0035] A computing power evaluation module, used for broadcasting the first computing task to the plurality of computing nodes, and calculating a computing power score of each computing node according to a time duration for each computing node to process the first computing task;

[0036] A node determination module is used to determine a target computing node to be assigned a computing task according to the computing power score of the computing node;

[0037] A task scheduling module is used to schedule the second computing task to the target computing node according to the computing power score.

[0038] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0039] processor;

[0040] a memory for storing instructions executable by the processor;

[0041] The processor is configured to execute the instructions to implement the task scheduling method as described above.

[0042] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the task scheduling method described above are implemented.

[0043] The task scheduling method provided by the embodiment of the present invention obtains the pending computing tasks cached by the scheduling node, broadcasts the first computing task to multiple computing nodes, calculates the computing power score of each computing node according to the time it takes each computing node to process the first computing task, determines the target computing node to which the computing task is to be assigned according to the computing power score of the computing node, and schedules the second computing task to the target computing node according to the computing power score. The embodiment of the present invention uniformly evaluates the computing power capabilities of computing nodes, clarifies the computing power characteristics of different computing nodes, timely intervenes in task allocation according to the computing power capabilities of different computing nodes, schedules computing tasks to computing nodes with sufficient computing power, realizes effective task scheduling for various types of computing nodes, and improves the effective utilization rate of computing power resources of computing nodes in heterogeneous clusters.

[0044] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0046] Figure 1 is a flowchart of a task scheduling method provided by an embodiment of the present invention;

[0047] Figure 2 yes Figure 1 A flowchart of step 102 in a task scheduling method provided by an embodiment of the present invention;

[0048] Figure 3 yes Figure 1A flowchart of step 104 in a task scheduling method provided by an embodiment of the present invention;

[0049] Figure 4 is a flowchart of another task scheduling method provided by an embodiment of the present invention;

[0050] Figure 5 is a flowchart of another task scheduling method provided by an embodiment of the present invention;

[0051] Figure 6 is a schematic diagram of the architecture of the task scheduling method provided by an embodiment of the present invention;

[0052] Figure 7 is a scenario schematic diagram of a task scheduling method provided by an embodiment of the present invention;

[0053] Figure 8 is a structural diagram of a task scheduling device provided by an embodiment of the present invention;

[0054] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0056] Reference Figure 1 , shows a flowchart of the steps of a task scheduling method provided by an embodiment of the present invention, which is applied to a scheduling cluster, wherein the scheduling cluster includes a scheduling node and a plurality of computing nodes, and the scheduling node communicates with the plurality of computing nodes. The method may include:

[0057] Step 101, obtaining the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task.

[0058] In an embodiment of the present invention, in order to solve the problem that different nodes are not considered when allocating tasks, it is difficult to take into account the processing capabilities of other types of nodes, resulting in the inability to meet the task allocation needs of heterogeneous clusters, this embodiment performs a unified computing power evaluation on various types of computing nodes managed by the cluster, and timely intervenes according to the available computing power of different computing nodes, schedules computing tasks to specific computing nodes, and realizes effective task scheduling for various types of computing nodes.

[0059] Specifically, the scheduling cluster first obtains the pending computing tasks cached by the scheduling node, wherein the pending computing tasks are sent to the scheduling node by the task source, and the scheduling node caches the pending computing tasks, wherein the pending computing tasks include a first computing task and a second computing task, the first computing task is a test task for evaluating the computing power capabilities of different computing nodes, and the second computing task is a task that each computing node needs to perform to meet the task request.

[0060] The embodiment of the present invention is applied to a scheduling cluster, the scheduling cluster includes a scheduling node and multiple computing nodes, the scheduling node communicates with the multiple computing nodes, and the functional units on the multiple computing nodes and one scheduling node are combined to complete the Figure 6 , showing the architecture diagram of the task scheduling method provided by an embodiment of the present invention, each computing node is bound to a fixed computing hardware, such as CPU, GPU or NPU, the scheduling node and the computing node can run on the same or different computers, requiring the scheduling node and the computing node to communicate with each other, and the scheduling node and the computing node together form a complete scheduling cluster. This embodiment can provide a unified task scheduler on the scheduling node to coordinate these resources when processing large-scale data tasks. The task scheduler not only needs to ensure that the tasks can be reasonably allocated and efficiently executed, but also takes into account that the nodes in the cluster may have different available computing power, and uses the available computing power of the current node as an important indicator for scheduling resources.

[0061] It should be noted that the embodiment of the present invention is applied to the scheduling cluster to process computing task requests, and its operation process may include one or more complete computing cycles. A complete computing cycle can be divided into a performance evaluation phase and a stable execution phase. The performance evaluation phase performs a basic computing power evaluation on the nodes managed by the cluster and experimentally schedules the tasks; the stable execution phase continuously schedules the tasks based on the evaluation results of the performance evaluation phase. In the performance evaluation phase, the computing nodes can be evaluated according to the computing power capabilities of different nodes to ensure the computing power utilization of the cluster; in the stable operation phase, timely intervention can be made according to the available computing power of different nodes to avoid excessive use of the computing power of the computing nodes.

[0062] Step 102, broadcast the first computing task to multiple computing nodes, and calculate the computing power score of each computing node according to the time it takes each computing node to process the first computing task.

[0063] In an embodiment of the present invention, a first computing task is broadcast to multiple computing nodes, and a computing power score of each computing node is calculated based on the time it takes for each computing node to process the first computing task. Specifically, the scheduling node is responsible for storing all task requests. When there is a scheduling task, the first computing task is first broadcast to all computing nodes in the scheduling cluster. Each computing unit will execute the same computing task. In one task scheduling, the next task scheduling will not be performed until all computing nodes have processed the currently scheduled computing task, so as to ensure the accuracy of the performance evaluation results of the computing nodes. Specifically, based on the time it takes for each computing node to process the first computing task, that is, the maximum completion time and average completion time of each computing node when processing the same computing task, the computing power capabilities of the computing nodes are statistically evaluated to obtain the computing power score of each computing node.

[0064] Step 103: Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node.

[0065] Step 104: Schedule the second computing task to the target computing node according to the computing power score.

[0066] Reference Figure 7 , showing a scenario diagram of the task scheduling method provided by an embodiment of the present invention. In the embodiment of the present invention, the scheduling cluster will rank all available computing nodes by computing power scores. The higher the computing power score, the stronger the computing power. After the performance evaluation phase is over, the scheduling cluster will enter the stable operation phase. In the stable operation phase, the scheduling node is responsible for storing all task requests. Each computing node will be scheduled for computing tasks in proportion to the computing power score. The tasks will be scheduled based on the target computing node determined by the above computing power score and the number of tasks that the target computing node can receive. Each computing node will be scheduled for computing tasks in proportion to the computing power score.

[0067] The task scheduling method provided by the embodiment of the present invention obtains the pending computing tasks cached by the scheduling node, broadcasts the first computing task to multiple computing nodes, calculates the computing power score of each computing node according to the time it takes each computing node to process the first computing task, determines the target computing node to which the computing task is to be assigned according to the computing power score of the computing node, and schedules the second computing task to the target computing node according to the computing power score. The embodiment of the present invention uniformly evaluates the computing power capabilities of computing nodes, clarifies the computing power characteristics of different computing nodes, timely intervenes in task allocation according to the computing power capabilities of different computing nodes, schedules computing tasks to computing nodes with sufficient computing power, realizes effective task scheduling for various types of computing nodes, and improves the effective utilization rate of computing power resources of computing nodes in heterogeneous clusters.

[0068] Further, see Figure 2 , showing Figure 1 A flowchart of step 102 in a task scheduling method is provided. The method is substantially the same as the task scheduling method provided in the first embodiment of the present invention. Step 102 may include:

[0069] Step 1021, in response to a computing power evaluation request for a computing node, broadcast a first computing task to multiple computing nodes.

[0070] Step 1022, obtaining the maximum completion time and average completion time of the computing node processing the first computing task.

[0071] Step 1023, using the maximum completion time and the average completion time to calculate the computing power score of each computing node.

[0072] It should be noted that in an embodiment of the present invention, when the scheduling cluster is started, the scheduling node will automatically trigger a computing power evaluation request. For example, when a hardware change occurs to a computing node in the cluster (such as adding or removing a node), or when the node performance needs to be re-evaluated after the cluster has run for a period of time, the scheduling node will also actively initiate a computing power evaluation request.

[0073] Specifically, the scheduling node sends the first computing task to all computing nodes through a broadcast mechanism. The broadcast mechanism ensures that each computing node can receive the same task, thereby ensuring the fairness and consistency of the evaluation. The broadcast can be carried out through a high-speed network (such as InfiniBand, Ethernet, etc.) to ensure that the computing task can be distributed to all computing nodes in a short time. The scheduling node is responsible for storing the tasks to be evaluated. After receiving the computing power evaluation request, it takes out the first computing task and broadcasts it to all computing nodes.

[0074] In this embodiment, after processing the first computing task, each computing node will report the completion time of the task to the scheduling node. The scheduling node collects the completion time data of all computing nodes. For each computing node, the scheduling node will calculate the maximum completion time and average completion time of all tasks processed by it, where the maximum completion time reflects the performance of the node when processing the most time-consuming task, and the average completion time reflects the overall performance of the node when processing multiple tasks. The two indicators of maximum completion time and average completion time can be used to comprehensively evaluate the computing power capacity of the computing node, taking into account both the peak performance of the node and its overall performance. Based on the maximum completion time and the average completion time, the scheduling node calculates the computing power score of each node.

[0075] The embodiment of the present invention distributes the first computing task to all computing nodes through a broadcast mechanism to ensure the fairness and consistency of the evaluation, collects the maximum completion time and average completion time of each computing node processing tasks, comprehensively reflects the computing power capabilities of the nodes, and obtains the computing power score of each node. The calculation of the computing power score provides an efficient decision-making basis for task scheduling and improves scheduling efficiency.

[0076] Specifically, step 1023 uses the maximum completion time and the average completion time to calculate the computing power score of each computing node, which may specifically include the following steps:

[0077] First, the maximum completion time and the average completion time are standardized, and a comprehensive score of the computing node is obtained according to the standardized maximum completion time and the average completion time;

[0078] Secondly, the comprehensive scores of the computing nodes are rounded and converted to obtain the computing power scores of the computing nodes.

[0079] It should be noted that, in the embodiment of the present invention, the computing node normalizes the maximum completion time and the average completion time for processing the first computing task, obtains the comprehensive score of the computing node based on the maximum completion time and the average completion time after normalization, rounds the comprehensive score of the computing node to obtain the computing power score of each computing node, and then reports the computing power score of each computing node to the scheduling node so that the scheduling node can clearly understand the computing power capacity of the computing node.

[0080] Specifically, the maximum completion time is the maximum completion time of all tasks processed by each computing node, and the average completion time is the average completion time of all tasks processed by each computing node. The maximum completion time and the average completion time can be calculated by the following formula:

[0081] C max，i =max(C i1 , C i2, ..., C in )

[0082]

[0083] Among them C max，i F is the maximum completion time of computing tasks processed by computing nodes. i is the average completion time of computing tasks processed by computing nodes, Cij represents the completion time of task j processed by node i, and n is the number of tasks processed by node i.

[0084] In this embodiment, in order to make the maximum completion time and the average completion time have the same scale, they can be standardized:

[0085]

[0086] Among them, Z max，i is the standard value of the maximum completion time, Z F，i is the standard value of the average completion time, μ max and σ max are the mean and standard deviation of the maximum completion time of all computing nodes, μ F and σ F are the mean and standard deviation of the average completion time of all computing nodes, respectively.

[0087] In this embodiment, the comprehensive score of each computing node is calculated based on the standardized maximum completion time and average completion time. In order to obtain an integer score, the comprehensive score can be rounded. The calculation formulas for the comprehensive score and the computing power score are as follows:

[0088] S i =ω1·Z max，i +ω2·Z F,i

[0089]

[0090] Among them, S i is the comprehensive score of the calculated node, ω1 and ω2 are weights used to indicate the importance of the maximum completion time and the average completion time in the score, S int，i is the computing power score of the computing node, a is the conversion coefficient, and Z max，i is the standard value of the maximum completion time, Z F,i is the standard value of the average completion time.

[0091] It should be noted that weights can be allocated according to actual needs. For example, ω1=ω2=0.5 means that the weights of the two are equal. This embodiment does not limit the specific values ​​of the weights. The reciprocal of the comprehensive score of the calculation node is because the shorter the time taken, the stronger the computing power of the node. Multiplying by the conversion coefficient facilitates the conversion of the value into an integer, which depends on the required integer score range and accuracy.

[0092] Further, see Figure 3 , showing Figure 1 A flowchart of step 104 in a task scheduling method is provided. The method is substantially the same as the task scheduling method provided in the first embodiment of the present invention. Step 104 may include:

[0093] Step 1041, determining the target computing node to which the computing task is to be assigned according to the computing power score of the computing node.

[0094] Step 1042: Determine the number of tasks to be allocated to the target computing node according to the computing power score of the target computing node.

[0095] Step 1043: Store the second computing task into a preset task scheduling slot list according to the task quantity of the target computing node.

[0096] Step 1044: Schedule the second computing task in the preset task scheduling slot list to the target computing node.

[0097] It should be noted that, in the embodiment of the present invention, after the performance evaluation phase is over, the scheduling node will sort the nodes according to the computing power score of each computing node. The node with a higher computing power score has a higher priority, indicating that its computing power is stronger. The scheduling node sorts according to the computing power score and selects the node with a higher computing power score as the target computing node, wherein the selection of the target computing node can be based on the complexity and priority of the task to ensure that complex tasks are preferentially assigned to nodes with stronger computing power. For example, for tasks with higher complexity, the scheduling node will select node A with the highest computing power score as the target node; for tasks with lower complexity, node B or C with the second highest computing power score can be selected. Specifically, the scheduling node allocates the number of tasks in proportion to the computing power score of the target computing node. The higher the computing power score of the node, the more tasks are allocated; the lower the computing power score of the node, the fewer tasks are allocated.

[0098] It should be noted that the scheduling node maintains a task scheduling slot list for storing the second computing task to be assigned. Each slot corresponds to a target computing node, and the number of tasks in the slot is proportional to the computing power score of the target node. For example, assuming that the computing power score of the target computing node is 6:3:2, the number of tasks of A, B, and C are 60, 30, and 20 respectively. The slot of node A in the task scheduling slot list stores 60 tasks, the slot of node B stores 30 tasks, and the slot of node C stores 20 tasks. The scheduling node stores the second computing task in the corresponding slot according to the number of tasks of the target node, and schedules the second computing task in the preset task scheduling slot list to the target computing node. The task storage can adopt a queue or list structure to ensure that the tasks can be assigned to the target computing node in sequence. The computing tasks to be assigned will enter the task scheduling slot list in sequence to ensure the orderliness of task assignment.

[0099] The embodiment of the present invention selects a node with a higher computing power score as the target computing node according to the computing power score of the computing node, ensures that the task can be assigned to the node with stronger computing power, allocates the number of tasks according to the computing power score ratio of the target node, maximizes the use of the computing power resources of the cluster, avoids resource waste, stores the second computing task in the task scheduling slot list according to the number of tasks of the target node, ensures the orderliness of task allocation, and schedules the tasks in the task scheduling slot list to the target computing node in sequence, ensuring that the tasks can be executed efficiently.

[0100] Specifically, in some embodiments, after step 103 determines the target computing node to be assigned the computing task according to the computing power score of the computing node, it also includes:

[0101] First, the computing power scores of the computing nodes are monitored and updated to determine computing nodes whose computing power scores are lower than a preset computing power threshold;

[0102] Secondly, the computing nodes whose computing power scores are lower than the preset computing power threshold are removed from the target computing nodes to be assigned computing tasks.

[0103] It should be noted that the scheduling node will recalculate the node's computing power score regularly (for example, every once in a while or after each task scheduling) to ensure that the computing power score can reflect the current status of the node. The scheduling node monitors the computing power score of the computing node in real time. The computing power score is updated based on the latest task processing data and resource usage to ensure the accuracy and real-time nature of the evaluation results. The preset computing power threshold is used as a criterion for judging whether the node is suitable to continue to undertake tasks. The computing power threshold can be dynamically adjusted according to the actual needs of the cluster and the computing power capacity of the node. For example, assuming that the preset computing power threshold is 3, a computing node with a computing power score lower than 3 will be considered to have insufficient computing power and is not suitable to continue to undertake tasks.

[0104] Specifically, the scheduling node selects computing nodes whose computing power scores are lower than the preset computing power threshold according to the latest computing power scores, marks these computing nodes as "insufficient computing power", and removes them from the target computing nodes to be assigned computing tasks. By setting the computing power threshold, it is possible to effectively avoid assigning tasks to nodes with insufficient computing power, ensure that tasks can be executed efficiently, and avoid task failures or delays due to insufficient node computing power. For tasks that have been assigned to nodes with insufficient computing power, the scheduling node will reinsert these tasks into the queue head of the task buffer unit, waiting to be reallocated to other nodes with sufficient computing power.

[0105] The embodiment of the present invention can monitor and update the computing power scores of computing nodes in real time, remove computing nodes whose computing power scores are lower than a preset computing power threshold from the target computing node list, ensure that tasks can be assigned to nodes with sufficient computing power, maximize the execution efficiency of tasks, avoid assigning tasks to nodes with insufficient computing power, ensure that computing power resources are maximized, and avoid waste of resources.

[0106] Reference Figure 4 , shows a flowchart of another task scheduling method provided by an embodiment of the present invention, which is basically the same as the task scheduling method provided by the first embodiment of the present invention, except that the method may further include:

[0107] Step 101, obtaining the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task.

[0108] Step 102, broadcast the first computing task to multiple computing nodes, and calculate the computing power score of each computing node according to the time it takes each computing node to process the first computing task.

[0109] Step 103: Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node.

[0110] The above steps 101 to 103 are described in the foregoing description and will not be repeated here.

[0111] Step 105, determine the available computing power of the target computing node, judge whether the available computing power of the target computing node meets the number of tasks of the target computing node, and identify the abnormal computing node.

[0112] Step 106, monitoring whether the available computing power of the abnormal computing node is restored within the preset resource-restricted time period.

[0113] Step 107: If the available computing power of the abnormal computing node is not restored within the preset resource restriction time, the target computing node to be assigned the computing task is re-determined.

[0114] It should be noted that in an embodiment of the present invention, the scheduling node monitors the available computing power of the target computing node in real time, wherein the evaluation indicators of the available computing power of the computing node include the utilization rate of the CPU, GPU / NPU, memory / video memory usage, etc., and the available computing power of each target computing node is obtained according to the current computing power resource usage of the node. The number of tasks of the target computing node is compared with the available computing power to determine whether the available computing power of the target computing node can meet its number of tasks. If the available computing power of the target computing node is not enough to meet its number of tasks, the scheduling node will mark the node as an abnormal computing node. The abnormal computing node will be deemed to be temporarily unable to undertake tasks, and subsequent tasks will no longer be assigned to the node.

[0115] It should be noted that the scheduling node will preset a resource restriction duration, which is used to determine whether the available computing power of the abnormal computing node can be restored within this time period. The resource restriction duration can be dynamically adjusted according to the actual operation of the cluster, and is usually set to a few minutes to tens of minutes. This embodiment does not make specific limitations on this.

[0116] Specifically, the scheduling node will continuously monitor the available computing power of the abnormal computing node to determine whether it has recovered to normal levels within the resource-constrained duration. If the available computing power of the abnormal computing node has recovered to normal levels within the resource-constrained duration, the scheduling node will re-mark it as a normal node and re-add it to the target computing node list. If the available computing power of the abnormal computing node has not recovered within the resource-constrained duration, the scheduling node will re-determine the target computing node to which the computing task is to be assigned.

[0117] The embodiment of the present invention monitors the available computing power of the target computing node in real time, determines whether it can meet the number of tasks, identifies abnormal computing nodes, monitors whether the available computing power of the abnormal computing node is restored within a preset resource-restricted time, and ensures that the node can continue to undertake tasks after the computing power is restored. If the available computing power of the abnormal computing node is not restored, the target computing node for the computing task to be assigned is re-determined to ensure that the task can be assigned to a node with sufficient computing power.

[0118] Compared with the prior art, the implementation mode of the present invention, on the basis of achieving the beneficial effects brought by the first implementation mode, the task reallocation mechanism can dynamically adjust the task allocation, ensure that the tasks can be timely allocated to nodes with sufficient computing power, maximize the execution efficiency of the tasks, and avoid frequent task reallocation caused by short-term fluctuations in node computing power resources by setting the resource restricted time, thereby ensuring the stability of task scheduling.

[0119] Reference Figure 5, shows a step flow chart of another task scheduling method provided by an embodiment of the present invention, which is basically the same as the task scheduling method provided by the first embodiment of the present invention, except that the method may further include:

[0120] Step 101, obtaining the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task.

[0121] Step 102, broadcast the first computing task to multiple computing nodes, and calculate the computing power score of each computing node according to the time it takes each computing node to process the first computing task.

[0122] Step 103: Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node.

[0123] Step 104: Schedule the second computing task to the target computing node according to the computing power score.

[0124] The above steps 101 to 104 are described in the foregoing description and will not be repeated here.

[0125] Step 108, monitor the remaining computing power of the target computing node.

[0126] Step 109, judging whether the target computing node exceeds the limit according to the preset computing resource upper limit and the remaining computing power of the target computing node.

[0127] Step 110: If yes, report the excess computing node to the scheduling node, and stop allocating computing tasks to the excess computing node.

[0128] It should be noted that in the embodiment of the present invention, during the process of the computing node stably executing the task, the scheduling node monitors the remaining computing power of the target computing node in real time. The evaluation of the remaining computing power is based on the current computing power resource usage of the node, including the utilization rate of the CPU, GPU / NPU, memory / video memory utilization rate, etc. The scheduling node will regularly collect the computing power resource data of the target computing node and calculate the remaining computing power of each node. The remaining computing power can be determined by the difference between the preset computing power resource upper limit of the computing node and the current computing power resource usage. The monitoring of the remaining computing power can help the scheduling node to reasonably allocate tasks, avoid allocating tasks to nodes with insufficient remaining computing power, and ensure that the computing power resources are maximized.

[0129] Specifically, the scheduling node will set a preset computing power resource upper limit for each target computing node as a standard for judging whether the node is over-limited. The preset computing power resource upper limit can be dynamically adjusted according to the hardware configuration of the node and the task requirements. The scheduling node will compare the remaining computing power of the target computing node with the preset computing power resource upper limit to determine whether the node is over-limited. If the remaining computing power of the target computing node is close to or lower than 0%, the scheduling node will consider the node to be over-limited and mark it as an over-limited computing node. If the target computing node is judged to be over-limited, the scheduling node will mark the node as an over-limited computing node and report the over-limit information to the scheduling node. The scheduling node will immediately stop allocating tasks to the over-limited computing node to avoid allocating new tasks to the over-limited node, prevent the node from crashing or task failure due to over-limited computing power. For tasks that have been assigned to the over-limited computing node, the scheduling node will reinsert these tasks into the queue head of the task buffer unit, waiting to be re-assigned to other nodes with sufficient computing power.

[0130] For example, to monitor whether the computing power resources on the node are over-limited, you need to pay special attention to the computing power load and usage of each node to ensure that the assigned tasks do not exceed the computing power limit of the node. Define the over-limit conditions and determine the over-limit standards of computing power resources to clarify when the over-limit alarm needs to be triggered. Common over-limit conditions include: CPU utilization, set the upper limit of CPU utilization, such as 80%-90%; GPU / NPU utilization, define the upper limit of GPU / NPU load and video memory utilization. Establish a real-time monitoring system to continuously track the usage of node computing power. Specifically, you can use system resource monitoring tools such as Prometheus, Node Exporter, etc. to collect CPU and GPU or NPU utilization in real time. Over-limit detection is used to analyze the collected computing power data in real time to determine whether it exceeds the preset threshold. Specifically, the following methods can be adopted: at regular intervals, check whether the computing power of each node has reached the set threshold, set resource overlimit events, and trigger detection once the computing power utilization of the node reaches the threshold. When the computing power resources of the computing node exceed the limit, an alarm is immediately triggered, and the alarm notifies the scheduling node to avoid allocating new tasks to the overlimit node.

[0131] Compared with the prior art, the implementation mode of the present invention, on the basis of achieving the beneficial effects brought by the first implementation mode, ensures that the scheduling node can obtain the computing power resource status of the node in time by real-time monitoring of the remaining computing power of the target computing node, avoids allocating tasks to nodes with insufficient remaining computing power, and avoids node crashes due to excessive computing power by timely identifying and processing over-limited computing nodes, thereby ensuring the stability of task scheduling.

[0132] Reference Figure 8, shows a schematic diagram of the structure of a task scheduling device provided by an embodiment of the present invention, which is applied to a scheduling cluster, wherein the scheduling cluster includes a scheduling node and a plurality of computing nodes, wherein the scheduling node communicates with the plurality of computing nodes, and the device includes:

[0133] The task acquisition module 201 is used to acquire the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task;

[0134] A computing power evaluation module 202 is used to broadcast the first computing task to the plurality of computing nodes, and calculate a computing power score of each computing node according to the time taken by each computing node to process the first computing task;

[0135] A node determination module 203 is used to determine a target computing node to be assigned a computing task according to the computing power score of the computing node;

[0136] The task scheduling module 204 is used to schedule the second computing task to the target computing node according to the computing power score.

[0137] Furthermore, the computing power evaluation module 202 includes:

[0138] A task broadcast submodule, configured to broadcast the first computing task to a plurality of computing nodes in response to a computing power evaluation request for the computing node;

[0139] An acquisition submodule, used to acquire a maximum completion time and an average completion time of the computing node processing the first computing task;

[0140] The computing power submodule is used to calculate the computing power score of each computing node using the maximum completion time and the average completion time.

[0141] Furthermore, the computing power submodule includes:

[0142] A processing unit, configured to perform normalization processing on the maximum completion time and the average completion time, and obtain a comprehensive score of the computing node according to the normalized maximum completion time and the average completion time;

[0143] The conversion unit is used to round the comprehensive scores of the computing nodes to obtain the computing power scores of the computing nodes.

[0144] Furthermore, the task scheduling module 204 includes:

[0145] A first determination submodule is used to determine a target computing node to be assigned a computing task according to a computing power score of the computing node;

[0146] A second determination submodule is used to determine the number of tasks to be allocated to the target computing node according to the computing power score of the target computing node;

[0147] A storage submodule, used for storing the second computing task into a preset task scheduling slot list according to the task quantity of the target computing node;

[0148] The scheduling submodule is used to schedule the second computing task in the preset task scheduling slot list to the target computing node.

[0149] Furthermore, the device also includes:

[0150] A first monitoring node module is used to monitor and update the computing power scores of the computing nodes, and determine the computing nodes whose computing power scores are lower than a preset computing power threshold;

[0151] The node removal module is used to remove the computing nodes whose computing power scores are lower than a preset computing power threshold from the target computing nodes to be assigned computing tasks.

[0152] Furthermore, the device also includes:

[0153] An abnormality judgment module is used to determine the available computing power of the target computing node, determine whether the available computing power of the target computing node meets the number of tasks of the target computing node, and determine the abnormal computing node;

[0154] The computing power monitoring module is used to monitor whether the available computing power of the abnormal computing node is restored within a preset resource-restricted time period.

[0155] The node redetermining module is used to redetermine the target computing node to be assigned the computing task if the available computing power of the abnormal computing node is not restored within the preset resource restriction time.

[0156] Furthermore, the device also includes:

[0157] A second monitoring node module, used to monitor the remaining computing power of the target computing node;

[0158] An over-limit judgment module, used to judge whether the target computing node exceeds the limit according to the preset computing power resource upper limit of the target computing node and the remaining computing power;

[0159] The node reporting module is used to report the over-limit computing node to the scheduling node and stop allocating computing tasks to the over-limit computing node.

[0160] The task scheduling device provided by the embodiment of the present invention obtains the pending computing tasks cached by the scheduling node, broadcasts the first computing task to multiple computing nodes, calculates the computing power score of each computing node according to the time taken by each computing node to process the first computing task, determines the target computing node to be assigned the computing task according to the computing power score of the computing node, and schedules the second computing task to the target computing node according to the computing power score. The embodiment of the present invention uniformly evaluates the computing power capabilities of the computing nodes, clarifies the computing power characteristics of different computing nodes, timely intervenes in task allocation according to the computing power capabilities of different computing nodes, schedules computing tasks to computing nodes with sufficient computing power, realizes effective task scheduling for various types of computing nodes, and improves the effective utilization rate of computing power resources of computing nodes in heterogeneous clusters.

[0161] Reference Fig. 9 , an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0162] A processor 301, a memory 303 for storing processor executable instructions;

[0163] The processor 301 is configured to execute the instructions to implement the task scheduling method as described above:

[0164] Obtaining the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task;

[0165] Broadcasting the first computing task to the plurality of computing nodes, and calculating a computing power score of each computing node according to a time duration for each computing node to process the first computing task;

[0166] Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node;

[0167] The second computing task is dispatched to the target computing node according to the computing power score.

[0168] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0169] The communication interface is used for communication between the above terminal and other devices.

[0170] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0171] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0172] In another embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the task scheduling method described in any of the above embodiments is implemented.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0174] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0175] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0176] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A task scheduling method, characterized in that: Applied to a scheduling cluster, the scheduling cluster includes a scheduling node and a plurality of computing nodes, the scheduling node communicates with the plurality of computing nodes, the method includes: Obtaining the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task; Broadcasting the first computing task to the plurality of computing nodes, and calculating a computing power score of each computing node according to a time duration for each computing node to process the first computing task; Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node; The second computing task is dispatched to the target computing node according to the computing power score.

2. The method according to claim 1, characterized in that The broadcasting of the first computing task to the plurality of computing nodes, and calculating a computing power score of each computing node according to a time duration for each computing node to process the first computing task, includes: In response to a computing power evaluation request for the computing node, broadcasting the first computing task to a plurality of the computing nodes; Obtaining a maximum completion time and an average completion time of the computing node processing the first computing task; The maximum completion time and the average completion time are used to calculate the computing power score of each computing node.

3. The method according to claim 2, characterized in that The calculating of the computing power score of each computing node by using the maximum completion time and the average completion time includes: The maximum completion time and the average completion time are normalized, and a comprehensive score of the computing node is obtained according to the normalized maximum completion time and the average completion time; The comprehensive scores of the computing nodes are rounded and converted to integers to obtain the computing power scores of the computing nodes.

4. The method according to claim 1, characterized in that: The step of dispatching the second computing task to the target computing node according to the computing power score includes: Determine the target computing node to which the computing task is to be assigned according to the computing power score of the computing node; Determine the number of tasks to be assigned to the target computing node according to the computing power score of the target computing node; Storing the second computing task in a preset task scheduling slot list according to the task quantity of the target computing node; The second computing task in the preset task scheduling slot list is scheduled to the target computing node.

5. The method according to claim 1, characterized in that After determining the target computing node to be assigned the computing task according to the computing power score of the computing node, the method further includes: Monitor and update the computing power scores of the computing nodes, and determine computing nodes whose computing power scores are lower than a preset computing power threshold; The computing nodes whose computing power scores are lower than the preset computing power threshold are removed from the target computing nodes to be assigned computing tasks.

6. The method according to claim 1, characterized in that After determining the target computing node to be assigned the computing task according to the computing power score of the computing node, the method further includes: Determine the available computing power of the target computing node, judge whether the available computing power of the target computing node meets the number of tasks of the target computing node, and determine the abnormal computing node; Monitor whether the available computing power of the abnormal computing node is restored within a preset resource-restricted time period. If the available computing power of the abnormal computing node is not restored within the preset resource restriction time, the target computing node to which the computing task is to be assigned is re-determined.

7. The method according to claim 1, characterized in that After dispatching the second computing task to the target computing node according to the computing power score, the method further includes: Monitoring the remaining computing power of the target computing node; Determining whether the target computing node exceeds the limit according to the preset computing power resource upper limit of the target computing node and the remaining computing power; If so, the excess computing node is reported to the scheduling node, and the allocation of computing tasks to the excess computing node is stopped.

8. A task scheduling device, characterized in that: Applied to a scheduling cluster, the scheduling cluster includes a scheduling node and a plurality of computing nodes, the scheduling node communicates with the plurality of computing nodes, the device includes: An acquisition task module, used to acquire the pending computing tasks cached by the scheduling node; wherein the pending computing tasks include a first computing task and a second computing task; A computing power evaluation module, used for broadcasting the first computing task to the plurality of computing nodes, and calculating a computing power score of each computing node according to a time duration for each computing node to process the first computing task; A node determination module is used to determine a target computing node to be assigned a computing task according to the computing power score of the computing node; A task scheduling module is used to schedule the second computing task to the target computing node according to the computing power score.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to execute the instructions to implement the task scheduling method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the task scheduling method according to any one of claims 1 to 7 is implemented.

Images