Elastic scheduling method and device, equipment and storage medium
By monitoring the task queue and using preset expansion and reduction strategies, and optimizing resource scheduling with the prediction model, the problem of low cluster resource utilization in deep learning model training is solved, and efficient resource utilization is achieved.
Patent Information
- Application Number
- CN202411144144.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2025-01-07
AI Technical Summary
During the training process of deep learning model, the problem of insufficient cluster resources during peak periods and wasted during low peak periods leads to low resource utilization.
By monitoring the task queue, using preset expansion and reduction strategies, using prediction models to predict idle resources, formulating expansion and reduction plans for flexible task nodes, and optimizing cluster resource scheduling.
This improves the utilization rate of cluster resources, avoids resource waste, and ensures the normal operation of model training tasks.
Smart Images
Figure CN119271391A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of flexible training scheduling, and in particular to a flexible scheduling method, device, equipment and storage medium. Background Art
[0002] With the rapid development of artificial intelligence technology, more and more companies have introduced a large number of deep learning models in their respective businesses to process massive data. Deep learning models need to be trained before they go online. For this reason, many companies using deep learning algorithms will build dedicated cluster resources for model training. Cluster resources will have peak and off-peak periods due to the amount of user use. During peak periods, resources may be insufficient, and the training tasks submitted by users need to be queued and scheduled in an orderly manner according to priority. During off-peak periods, cluster resources will be idle, resulting in resource waste. Summary of the invention
[0003] The main purpose of the present invention is to provide a flexible scheduling method, device, equipment and storage medium, aiming to solve the technical problem of how to improve cluster resource utilization during model training.
[0004] A first aspect of the present invention provides a flexible scheduling method, the flexible scheduling method comprising:
[0005] Monitor whether the task queues corresponding to each task node in the current cluster are empty;
[0006] If there is an empty task queue, a capacity expansion plan corresponding to the elastic task node is formulated based on the preset capacity expansion strategy and cluster resource scheduling is performed;
[0007] If there is a non-empty task queue currently, determine whether the cluster resources corresponding to the current non-empty task queue meet the current non-empty task queue;
[0008] If the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, a scaling-down plan corresponding to the elastic task node is formulated based on the preset scaling-down strategy and cluster resource scheduling is performed.
[0009] Optionally, in a first implementation of the first aspect of the present invention, if there is currently an empty task queue, formulating a capacity expansion plan corresponding to the elastic task node based on a preset capacity expansion strategy and performing cluster resource scheduling includes:
[0010] If there is an empty task queue, the trained prediction model is used to predict the idle resources of the task nodes corresponding to the empty task queue to obtain the idle resources in the current cluster.
[0011] Based on the idle resources in the current cluster, the capacity expansion of elastic task nodes is calculated and multiple expansion plans are obtained;
[0012] Score each expansion plan respectively, and use the expansion plan with the highest score as the expansion plan adopted by the elastic task node;
[0013] Based on the final selected expansion plan, cluster resources are scheduled for the elastic task nodes to be expanded.
[0014] Optionally, in a second implementation of the first aspect of the present invention, the prediction model includes a CPU prediction model, a memory prediction model, and a GPU prediction model, and the prediction model is obtained by the following training method:
[0015] Obtain monitoring data of different types of historical resources for each task node in the cluster;
[0016] Select each target time point within the preset time length in turn, perform minute-level aggregation on the monitoring data of each target time point N days before each target time point, obtain a time series of the first length and use it as model input training data of the corresponding prediction model;
[0017] Select each target time point within the preset time length in turn, perform minute-level aggregation on the monitoring data of each target time point M days after each target time point, obtain a time series of the second length and use it as model output training data of the corresponding prediction model;
[0018] Based on various types of model input training data, model output training data and preset loss functions, different types of prediction models are trained respectively to obtain different types of prediction models trained by each task node in the cluster.
[0019] Optionally, in a third implementation of the first aspect of the present invention, the expansion calculation of the elastic task node based on the idle resources in the current cluster to obtain multiple expansion solutions includes:
[0020] Get the amount of resources used by the elastic task node for single-copy training;
[0021] Taking the amount of resources used by the single replica as the expansion benchmark, based on the idle resources in the current cluster, the expansion of the elastic task node is measured and calculated to obtain multiple expansion plans;
[0022] Each of the expansion plans includes the amount of resources used for the expansion and the expansion operation time, and the expansion operation time exceeds the task scheduling period of the elastic task node.
[0023] Optionally, in a fourth implementation of the first aspect of the present invention, scoring each expansion plan respectively and using the expansion plan with the highest score as the expansion plan adopted by the elastic task node includes:
[0024] Based on the amount of resources used for capacity expansion and the running time of capacity expansion in each of the capacity expansion plans, the score of each capacity expansion plan is calculated, and the capacity expansion plan with the highest score is used as the capacity expansion plan adopted by the elastic task node;
[0025] The scoring formula is: S=F*T, where S represents the score value, F represents the amount of resources used for capacity expansion, and T represents the duration of capacity expansion operation.
[0026] Optionally, in a fifth implementation of the first aspect of the present invention, if there is currently a non-empty task queue, determining whether the current cluster resources satisfy the current non-empty task queue includes:
[0027] If there is a non-empty task queue, the available resources of the cluster resources corresponding to the current non-empty task queue and the resources required by the tasks to be scheduled in the current non-empty queue are counted;
[0028] If the available resource amount of the cluster resource corresponding to the current non-empty task queue is greater than or equal to the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource satisfies the current non-empty task queue;
[0029] If the available resource amount of the cluster resource corresponding to the current non-empty task queue is less than the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource does not satisfy the current non-empty task queue.
[0030] Optionally, in a sixth implementation of the first aspect of the present invention, if the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, then based on a preset scaling-down strategy, formulating a scaling-down plan corresponding to the elastic task node and performing cluster resource scheduling includes:
[0031] If the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, count whether the amount of resources available for scaling down the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue;
[0032] If the amount of resources available for scaling down of the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue, then based on the amount of resources required by the tasks to be scheduled in the current non-empty queue, the scaling down of the adapted elastic task node is calculated to obtain a scaling down plan;
[0033] Based on the scaling-down solution, cluster resource scheduling is performed on the elastic task nodes to be scaled down.
[0034] A second aspect of the present invention provides an elastic scheduling device, the elastic scheduling device comprising:
[0035] The monitoring module is used to monitor whether the task queue corresponding to each task node in the current cluster is empty;
[0036] The capacity expansion module is used to formulate a capacity expansion plan corresponding to the elastic task node and perform cluster resource scheduling based on the preset capacity expansion strategy if there is an empty task queue at present;
[0037] A judgment module is used to judge whether the cluster resources corresponding to the current non-empty task queue satisfy the current non-empty task queue if there is a non-empty task queue currently;
[0038] The scaling-down module is used to formulate a scaling-down plan corresponding to the elastic task node and perform cluster resource scheduling based on a preset scaling-down strategy if the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue.
[0039] Optionally, in a first implementation of the second aspect of the present invention, the capacity expansion module includes:
[0040] The prediction unit is used to use the trained prediction model to predict the idle resources of the task nodes corresponding to the empty task queue if there is an empty task queue currently, so as to obtain the idle resources in the current cluster;
[0041] The capacity expansion calculation unit is used to calculate the capacity expansion of elastic task nodes based on the idle resources in the current cluster and obtain multiple capacity expansion plans;
[0042] A scoring unit is used to score each expansion plan respectively, and use the expansion plan with the highest score as the expansion plan adopted by the elastic task node;
[0043] The capacity expansion scheduling unit is used to schedule cluster resources for the elastic task nodes to be expanded based on the final selected capacity expansion plan.
[0044] Optionally, in a second implementation of the second aspect of the present invention, the prediction model includes a CPU prediction model, a memory prediction model, and a GPU prediction model, and the elastic scheduling device further includes:
[0045] The training module is used to obtain monitoring data of different types of historical resources of each task node in the cluster;
[0046] Select each target time point within the preset duration in turn, perform minute-level aggregation on the monitoring data of each target time point N days before each target time point, obtain a time series of the first length and use it as the model input training data of the corresponding prediction model; select each target time point within the preset duration in turn, perform minute-level aggregation on the monitoring data of each target time point M days after each target time point, obtain a time series of the second length and use it as the model output training data of the corresponding prediction model; based on various types of model input training data, model output training data and preset loss functions, train different types of prediction models respectively, and obtain different types of prediction models trained for each task node in the cluster.
[0047] Optionally, in a third implementation of the second aspect of the present invention, the measuring unit is specifically configured to:
[0048] Get the amount of resources used by the elastic task node for single-copy training;
[0049] Taking the amount of resources used by the single replica as the expansion benchmark, based on the idle resources in the current cluster, the expansion of the elastic task node is measured and calculated to obtain multiple expansion plans;
[0050] Each of the expansion plans includes the amount of resources used for the expansion and the expansion operation time, and the expansion operation time exceeds the task scheduling period of the elastic task node.
[0051] Optionally, in a fourth implementation of the second aspect of the present invention, the scoring unit is specifically configured to:
[0052] Based on the amount of resources used for capacity expansion and the running time of capacity expansion in each of the capacity expansion plans, the score of each capacity expansion plan is calculated, and the capacity expansion plan with the highest score is used as the capacity expansion plan adopted by the elastic task node;
[0053] The scoring formula is: S=F*T, where S represents the score value, F represents the amount of resources used for capacity expansion, and T represents the duration of capacity expansion operation.
[0054] Optionally, in a fifth implementation of the second aspect of the present invention, the judgment module is specifically used to:
[0055] If there is a non-empty task queue, the available resources of the cluster resources corresponding to the current non-empty task queue and the resources required by the tasks to be scheduled in the current non-empty queue are counted;
[0056] If the available resource amount of the cluster resource corresponding to the current non-empty task queue is greater than or equal to the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource satisfies the current non-empty task queue;
[0057] If the available resource amount of the cluster resource corresponding to the current non-empty task queue is less than the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource does not satisfy the current non-empty task queue.
[0058] Optionally, in a sixth implementation of the second aspect of the present invention, the capacity reduction module includes:
[0059] A statistical unit, used for counting whether the amount of resources available for scaling down of the current elastic task node is greater than the amount of resources required for the tasks to be scheduled in the current non-empty queue if the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue;
[0060] A capacity reduction calculation unit is used to calculate the capacity reduction of the adapted elastic task node based on the amount of resources required by the tasks to be scheduled in the current non-empty queue if the amount of resources available for reduction of the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue, and obtain a capacity reduction plan;
[0061] The capacity reduction scheduling unit is used to perform cluster resource scheduling on the elastic task nodes to be reduced based on the capacity reduction plan.
[0062] A third aspect of the present invention provides a computer device, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the computer device executes the above-mentioned flexible scheduling method.
[0063] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned flexible scheduling method.
[0064] The present invention provides an elastic scheduling method, device, equipment and storage medium. First, each task node in the current cluster is monitored in real time, so as to timely discover an empty task queue. An empty task queue indicates that there are idle resources in the corresponding cluster node, which can be used for elastic task node expansion. Therefore, based on the preset expansion strategy, an expansion plan corresponding to the elastic task node is formulated, and then cluster resource scheduling is performed to improve cluster resource utilization. In addition, in order to avoid the occupation of expansion resources by the elastic task node and affect the non-elastic training task, when there is a non-empty task queue in the cluster and the current cluster resources do not meet the current non-empty task queue, the elastic task node can be reduced in capacity, a reduction plan corresponding to the elastic task node is formulated, and the cluster resources generated by the reduction are used for non-elastic training tasks. The present invention can intelligently adjust the resource amount of the elastic training task according to the idle situation of the cluster resource amount, so as to more flexibly utilize the current cluster resources and improve the cluster resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 A schematic diagram of an embodiment of a flexible scheduling method in an embodiment of the present invention;
[0066] Figure 2 A schematic diagram of an embodiment of resource prediction by a prediction model in an embodiment of the present invention;
[0067] Figure 3 A schematic diagram of an embodiment of a flexible scheduling device in an embodiment of the present invention;
[0068] Figure 4 FIG. 1 is a schematic diagram of an embodiment of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION
[0069] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0070] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the flexible scheduling method in the embodiment of the present invention includes:
[0071] 101. Monitor whether the task queue corresponding to each task node in the current cluster is empty;
[0072] In this embodiment, the cluster used for model training is composed of multiple cluster nodes, each cluster node is a task node for model training, and each task node corresponds to a task queue. When a user submits a model training task to any cluster node (ie, task node), the model training task will be written into the corresponding task queue and wait for scheduling.
[0073] In this embodiment, in order to realize the real-time utilization of idle resources, it is necessary to monitor in real time whether the task queue corresponding to each task node is empty. An empty task queue means that there is no model training task to be executed, that is, there are idle cluster resources on the task node.
[0074] 102. If there is an empty task queue, a capacity expansion plan corresponding to the elastic task node is formulated based on the preset capacity expansion strategy and cluster resource scheduling is performed;
[0075] In this embodiment, if there are one or more empty task queues in the current cluster, the idle cluster resources (such as GPU, CPU, memory, etc.) of the corresponding task node can be used for model training of other elastic task nodes. Among them, the cluster resources used for model training in the elastic task node can be flexibly changed, such as resource expansion or resource reduction. In this embodiment, when there is an empty task queue, it is determined that expansion can be performed, and specifically based on the preset expansion strategy, the expansion plan corresponding to the elastic task node is formulated, and then the cluster resources are scheduled according to the expansion plan.
[0076] In one embodiment, the step 102 further includes:
[0077] 1021. If there is an empty task queue, use the trained prediction model to predict the idle resources of the task node corresponding to the empty task queue to obtain the idle resources in the current cluster;
[0078] In this embodiment, when an empty task queue is detected, in order to better utilize the idle resources of the cluster and avoid frequent expansion and contraction, this embodiment introduces a prediction model that can estimate the amount of idle resources and idle time of each node in the cluster in the future.
[0079] In this embodiment, the input of the prediction model is the idle resource amount of the task node in a certain period of time before the current time point (for example, 3 days), and the output is the idle resource amount of the task node in a certain period of time after the current time point (for example, one day). By predicting the idle resources of the task nodes corresponding to all empty task queues in the cluster, all the idle resources in the current cluster can be obtained.
[0080] For example, there are empty task queues in task nodes A, B, and C in the current cluster. The idle resources are predicted through the prediction models corresponding to each task node, and the idle resources are obtained respectively: the idle resource amount of task node A is X1, the idle resource amount of task node B is X2, and the idle resource amount of task node C is X3. Then the idle resource amount in the current cluster is X1+X2+X3. It should be noted that due to the different training conditions of each task node, for example, the idle time of resources of each cluster node is different, that is, the idle resource amount is time series data, and the total idle resource amount at each time point is different. Therefore, it is necessary to expand the capacity of the elastic task nodes to further improve the utilization rate of idle resources.
[0081] In one embodiment, considering that different task nodes may use different resource types, for example, task node A uses CPU resources for model training, task node B uses GPU resources for model training, and task node C uses memory resources for model training, therefore, in order to accurately predict the future idle resource situation of each task node, it is necessary to train corresponding prediction models for different resource types respectively, and the prediction models include CPU prediction model, memory prediction model and GPU prediction model.
[0082] In one embodiment, the prediction model in step 1021 is obtained by the following training method:
[0083] (1) Obtain monitoring data of different types of historical resources of each task node in the cluster;
[0084] In this embodiment, each task node in the cluster has a training state and a non-training state during operation. In the training state, the cluster node has no idle resources (that is, the idle resource amount is 0), and in the non-training state, the resources of the cluster node are in an idle state. The monitoring data of different types of historical resources of each task node in the cluster are obtained respectively.
[0085] (2) selecting each target time point within a preset time length in turn, and performing minute-level aggregation on the monitoring data of each target time point N days before each target time point, to obtain a time series of a first length and use it as model input training data for a corresponding prediction model;
[0086] (3) selecting each target time point within a preset time length in turn, and performing minute-level aggregation on the monitoring data of each target time point M days after each target time point, to obtain a time series of a second length and use it as model output training data of the corresponding prediction model;
[0087] In this embodiment, if the node reports monitoring data once per second, it is necessary to aggregate the monitoring data of each second at the minute level, so that the average value of multiple monitoring data within one minute is used as the monitoring data corresponding to the current minute time point, that is, one piece of monitoring data corresponds to each minute.
[0088] For example, task node A uses CPU resources for model training. After obtaining the corresponding monitoring data, the monitoring data is aggregated at the minute level: assuming that the selected number of days is N = 3 days, the idle resource amount per minute 3 days before time point a is: [(a-4320min,cpu), (a-4319min,cpu), …, (a-1min,cpu), (a,cpu)], that is, a time series with a length of 4320 is obtained, and the time series is used as a model input training data of the corresponding prediction model, where the idle resource amount cpu corresponding to each minute may be 0 or not 0. If it is 0, it means that there is no idle resource. Taking the time series data corresponding to a time point of one month as training data, a total of 30x24x60 = 43200 model input training data.
[0089] Similarly, assuming that the selected number of days is M = 1 day, the idle resource amount per minute from time point a to the next day is: [(a+1min,cpu), (a+2min,cpu), ..., (a+1440min,cpu)]), that is, a time series with a length of 1440 is obtained, and the time series is used as a model output training data of the corresponding prediction model, where the idle resource amount cpu corresponding to each minute may be 0 or not 0. If it is 0, it means that there is no idle resource. Taking the time series data corresponding to a time point of one month as training data, there are a total of 30x24x60=43200 model output training data.
[0090] (4) Based on various types of model input training data, model output training data, and preset loss functions, different types of prediction models are trained respectively to obtain different types of prediction models trained by each task node in the cluster.
[0091] In this embodiment, one task node of the cluster corresponds to one prediction model (CPU prediction model or GPU prediction model or memory prediction model), and each task node of the cluster needs to obtain a corresponding amount of training data (for example, 43,200 model input and output training data) for model training.
[0092] After obtaining the training data, the model is trained according to the back propagation principle, and the training data is input into the network. The data of the past day and the next day are used as the regression target for training, and the loss value is calculated using the mean square error function. The structure of the model can be freely set. In this embodiment, the Transformer model is preferably used (other models, such as recurrent neural networks and one-dimensional convolutional models, can also be used). The input sequence length is set to 4320, that is, the resource idleness of the past three days, and the output sequence length is 1440, that is, the resource idleness of the next day.
[0093] In this embodiment, different types of prediction models trained for each task node in the cluster can be obtained by training in the above manner, that is, each prediction model is only used to predict the idle resources of the task node.
[0094] 1022. Based on the idle resources in the current cluster, the capacity expansion of the elastic task nodes is calculated to obtain multiple capacity expansion plans;
[0095] In this embodiment, after obtaining the idle resources in the current cluster through the prediction model, it is necessary to further perform capacity expansion calculation on the elastic task nodes, and each capacity expansion calculation corresponds to a capacity expansion plan.
[0096] In one embodiment, the step 1022 further includes:
[0097] (1) Obtain the amount of resources used by the elastic task node for single-copy training;
[0098] (2) Taking the amount of resources used by the single replica as the expansion benchmark, based on the idle resources in the current cluster, the expansion of the elastic task node is measured and calculated to obtain multiple expansion plans;
[0099] Each of the expansion plans includes the amount of resources used for the expansion and the expansion operation time, and the expansion operation time exceeds the task scheduling period of the elastic task node.
[0100] In this embodiment, in order to improve resource utilization, the following strategy is preferably used for resource expansion measurement: first try to expand a single copy, then try to expand multiple copies in sequence, until the estimated runnable time does not meet the task scheduling cycle. Among them, the task scheduling cycle corresponds to a cycle of model training, that is, the duration of one iteration. Among them, the amount of resources used for single copy training will be pre-published before training and the amount of resources is fixed.
[0101] like Figure 2 As shown in the figure, assuming that the resource usage of a single replica training of elastic task node X is 1 GPU, the task scheduling cycle is 1 hour, and there are 3 empty task queues at the current time point t0, corresponding to task nodes A, B, and C respectively. The corresponding prediction models A, B, and C are used to predict idle resources, and the idle resource prediction results are:
[0102] Task node A: The idle resources are 2 GPUs, and the expansion runtime is (t1-t0)=1h;
[0103] Task node B: The idle resources are 2 GPUs, and the expansion runtime is (t2-t0)=2h;
[0104] Task node C: The idle resource is 1 GPU, and the expansion runtime is (t3-t0)=3h;
[0105] According to the above resource expansion strategy, taking the amount of resources used by a single replica as the expansion benchmark, the following expansion plan can be formed:
[0106] Expansion plan 1: expand the resource volume by 1 GPU and expand the running time by 1 hour;
[0107] Expansion plan 2: expand the resource volume by 2 GPUs and expand the running time by 1 hour;
[0108] Expansion plan 3: expand the resource volume by 3 GPUs and the expansion runtime by 1 hour;
[0109] Expansion plan 4: expand the resource volume to 4 GPUs and expand the running time to 1 hour;
[0110] Expansion plan 5: expand the resource volume to 5 GPUs and expand the running time to 1 hour;
[0111] Expansion plan 6: expand the resource volume by 2 GPUs and expand the running time by 2 hours;
[0112] Expansion Plan 7: Expand the resource volume by 3 GPUs and expand the running time by 2 hours;
[0113] Expansion Plan 8: Expand the resource volume by 1 GPU and expand the running time by 5 hours.
[0114] 1023. Score each expansion plan respectively, and use the expansion plan with the highest score as the expansion plan adopted by the elastic task node;
[0115] In this embodiment, since there are multiple capacity expansion plans, it is necessary to select the best capacity expansion plan through scoring as the capacity expansion plan to be adopted by the elastic task node.
[0116] In one embodiment, the above step 1023 specifically scores each expansion plan in the following manner:
[0117] Based on the amount of resources used for capacity expansion and the running time of capacity expansion in each of the capacity expansion plans, the score of each capacity expansion plan is calculated, and the capacity expansion plan with the highest score is used as the capacity expansion plan adopted by the elastic task node;
[0118] The scoring formula is: S=F*T, where S represents the score value, F represents the amount of resources used for capacity expansion, and T represents the duration of capacity expansion operation.
[0119] For example, continue with Figure 2 Taking as an example, based on the above scoring formula, we can get the scoring values corresponding to each expansion plan:
[0120] The score of expansion plan 1 is: 1*1=1;
[0121] The score of expansion plan 2 is: 2*1=2;
[0122] The score of expansion plan 3 is: 3*1=3;
[0123] The score of expansion plan 4 is: 4*1=4;
[0124] The score of expansion plan 5 is: 5*1=5;
[0125] The score of expansion plan 6 is: 2*2=4;
[0126] The score of expansion plan 7 is: 3*2=6;
[0127] The score of expansion plan 8 is: 1*5=5;
[0128] As can be seen from the above, expansion plan 7 has the highest score, so expansion plan 7 is selected as the expansion plan adopted by the elastic task node. It should be noted that if there are expansion plans with the same score, one can be randomly selected, or the expansion plan with the largest amount of expandable resources among the same score can be selected.
[0129] 1024. Based on the finally selected expansion plan, cluster resources are scheduled for the elastic task nodes to be expanded.
[0130] Continue with Figure 2For example, through the above scoring, it is finally determined that expansion plan 7 is selected as the expansion plan adopted by the elastic task node. Expansion plan 7 is to simultaneously schedule the idle resources of task nodes B and C for 2 hours each for elastic node training.
[0131] 103. If there is a non-empty task queue currently, determine whether the cluster resources corresponding to the current non-empty task queue satisfy the current non-empty task queue;
[0132] In this embodiment, if the elastic node adopts the expansion plan and occupies the idle resources of other task nodes, then when other task nodes receive new training tasks, it is possible that there will be insufficient resources for the current task to be trained. Therefore, in this embodiment, if there is a non-empty task queue, it is first determined whether the cluster resources corresponding to the current non-empty task queue meet the current non-empty task queue, so as to determine whether it is necessary to reduce the capacity of the elastic node that occupies the cluster resources corresponding to the non-empty task queue due to expansion.
[0133] In one embodiment, the above step 103 further includes:
[0134] 1031. If there is a non-empty task queue currently, the available resource amount of the cluster resource corresponding to the current non-empty task queue and the resource amount required by the tasks to be scheduled in the current non-empty queue are counted;
[0135] 1032. If the available amount of cluster resources corresponding to the current non-empty task queue is greater than or equal to the amount of resources required by the tasks to be scheduled in the current non-empty queue, it is determined that the current cluster resources satisfy the current non-empty task queue;
[0136] 1033. If the available resource amount of the cluster resource corresponding to the current non-empty task queue is less than the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource does not satisfy the current non-empty task queue.
[0137] In this embodiment, if there is currently a non-empty task queue, the available resource amount of the cluster resources corresponding to the current non-empty task queue and the resource amount required by the tasks to be scheduled in the current non-empty queue are counted.
[0138] For example, if there is a non-empty queue for task node S, the available resources of the cluster resources of task node S are first counted. For example, if there is 1 GPU left in task node S, and the resources required for the tasks to be scheduled in the current non-empty queue are 3 GPUs, then task node S has insufficient resources and cannot meet the resources required for the tasks to be scheduled in the current queue.
[0139] 104. If the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, a scaling-down plan corresponding to the elastic task node is formulated based on a preset scaling-down strategy and cluster resource scheduling is performed.
[0140] In this embodiment, if the cluster resources corresponding to the current non-empty task queue do not meet the needs of the current non-empty task queue, it is necessary to adopt a preset scaling-down strategy and formulate a scaling-down plan corresponding to the elastic task node, so as to schedule the cluster resources generated by the scaling-down of the elastic task node to the tasks to be scheduled in the non-empty queue, thereby ensuring the normal operation of the tasks to be scheduled in the non-empty queue.
[0141] In one embodiment, the step 104 further includes:
[0142] 1041. If the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, count whether the amount of resources available for scaling down of the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue;
[0143] 1042. If the amount of resources available for shrinking the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue, then based on the amount of resources required by the tasks to be scheduled in the current non-empty queue, shrinking calculation is performed on the adapted elastic task node to obtain a shrinking plan;
[0144] 1043. Based on the scaling-down plan, cluster resource scheduling is performed on the elastic task nodes to be scaled down.
[0145] In this embodiment, when specifying a scaling-down plan for an elastic task node, based on the preset scaling-down strategy, first count whether the amount of resources available for scaling down of the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue. A scaling-down plan can only be specified if the amount of resources available for scaling down of the elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue. If the amount of resources available for scaling down of the elastic task node is less than the amount of resources required by the tasks to be scheduled in the current non-empty queue, wait for other tasks to release resources or remind the tester to handle it, such as increasing cluster resources.
[0146] In this embodiment, considering that the elastic task also needs to be trained, it is necessary to shrink the capacity under the premise of ensuring the operation of the elastic task itself. For example, the current elastic task node uses 5 GPU resources for model training, and the minimum resources required for training are 2 GPUs. The amount of resources available for shrinking the elastic task node is 3 GPUs. If the amount of resources required for the tasks to be scheduled in the current non-empty queue is 2 GPUs, the shrinking solution obtained by shrinking the elastic task node is: shrink the elastic task node by 2 GPU resources and provide them to the tasks to be scheduled in the current non-empty queue.
[0147] In this implementation, each task node in the current cluster is first monitored in real time to promptly discover an empty task queue. An empty task queue indicates that there are idle resources in the corresponding cluster node, which can be used for elastic task node expansion. Therefore, based on the preset expansion strategy, an expansion plan corresponding to the elastic task node is formulated, and then cluster resource scheduling is performed to improve cluster resource utilization. In addition, in order to avoid the elastic task node occupying the expansion resources and affecting the non-elastic training tasks, when there is a non-empty task queue in the cluster and the current cluster resources do not meet the current non-empty task queue, the elastic task node can be scaled down, a scaling-down plan corresponding to the elastic task node can be formulated, and the cluster resources generated by the scaling-down can be used for non-elastic training tasks. This embodiment can intelligently adjust the resource amount of the elastic training task according to the idleness of the cluster resources, so as to more flexibly utilize the current cluster resources and improve the cluster resource utilization.
[0148] The above describes the flexible scheduling method in the embodiment of the present invention. The following describes the flexible scheduling device in the embodiment of the present invention. Figure 3 , an embodiment of the flexible scheduling device in the embodiment of the present invention includes:
[0149] Monitoring module 301, used to monitor whether the task queue corresponding to each task node in the current cluster is empty;
[0150] The capacity expansion module 302 is used to formulate a capacity expansion plan corresponding to the elastic task node and perform cluster resource scheduling based on a preset capacity expansion strategy if there is currently an empty task queue;
[0151] The judgment module 303 is used to judge whether the cluster resources corresponding to the current non-empty task queue satisfy the current non-empty task queue if there is a non-empty task queue currently;
[0152] The scaling-down module 304 is used to formulate a scaling-down plan corresponding to the elastic task node and perform cluster resource scheduling based on a preset scaling-down strategy if the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue.
[0153] Optionally, in one embodiment, the capacity expansion module 302 includes:
[0154] The prediction unit is used to use the trained prediction model to predict the idle resources of the task nodes corresponding to the empty task queue if there is an empty task queue currently, so as to obtain the idle resources in the current cluster;
[0155] The capacity expansion calculation unit is used to calculate the capacity expansion of elastic task nodes based on the idle resources in the current cluster and obtain multiple capacity expansion plans;
[0156] A scoring unit is used to score each expansion plan respectively, and use the expansion plan with the highest score as the expansion plan adopted by the elastic task node;
[0157] The capacity expansion scheduling unit is used to schedule cluster resources for the elastic task nodes to be expanded based on the final selected capacity expansion plan.
[0158] Optionally, in one embodiment, the prediction model includes a CPU prediction model, a memory prediction model, and a GPU prediction model, and the flexible scheduling device further includes:
[0159] The training module 305 is used to obtain the monitoring data of different types of historical resources of each task node in the cluster; select each target time point within the preset duration in turn, and perform minute-level aggregation on the monitoring data of each target time point N days before each target time point, so as to obtain a time series of a first length and use it as the model input training data of the corresponding prediction model; select each target time point within the preset duration in turn, and perform minute-level aggregation on the monitoring data of each target time point M days after each target time point, so as to obtain a time series of a second length and use it as the model output training data of the corresponding prediction model; based on each type of model input training data, model output training data and preset loss function, train different types of prediction models respectively, and obtain different types of prediction models trained for each task node in the cluster.
[0160] Optionally, in one embodiment, the measuring unit is specifically used for:
[0161] Get the amount of resources used by the elastic task node for single-copy training;
[0162] Taking the amount of resources used by the single replica as the expansion benchmark, based on the idle resources in the current cluster, the expansion of the elastic task node is measured and calculated to obtain multiple expansion plans;
[0163] Each of the expansion plans includes the amount of resources used for the expansion and the expansion operation time, and the expansion operation time exceeds the task scheduling period of the elastic task node.
[0164] Optionally, in one embodiment, the scoring unit is specifically used for:
[0165] Based on the amount of resources used for capacity expansion and the running time of capacity expansion in each of the capacity expansion plans, the score of each capacity expansion plan is calculated, and the capacity expansion plan with the highest score is used as the capacity expansion plan adopted by the elastic task node;
[0166] The scoring formula is: S=F*T, where S represents the score value, F represents the amount of resources used for capacity expansion, and T represents the duration of capacity expansion operation.
[0167] Optionally, in one embodiment, the judging module 303 is specifically configured to:
[0168] If there is a non-empty task queue, the available resources of the cluster resources corresponding to the current non-empty task queue and the resources required by the tasks to be scheduled in the current non-empty queue are counted;
[0169] If the available resource amount of the cluster resource corresponding to the current non-empty task queue is greater than or equal to the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource satisfies the current non-empty task queue;
[0170] If the available resource amount of the cluster resource corresponding to the current non-empty task queue is less than the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource does not satisfy the current non-empty task queue.
[0171] Optionally, in one embodiment, the capacity reduction module includes:
[0172] A statistical unit, used for counting whether the amount of resources available for scaling down of the current elastic task node is greater than the amount of resources required for the tasks to be scheduled in the current non-empty queue if the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue;
[0173] A capacity reduction calculation unit is used to calculate the capacity reduction of the adapted elastic task node based on the amount of resources required by the tasks to be scheduled in the current non-empty queue if the amount of resources available for reduction of the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue, and obtain a capacity reduction plan;
[0174] The capacity reduction scheduling unit is used to perform cluster resource scheduling on the elastic task nodes to be reduced based on the capacity reduction plan.
[0175] Since the embodiments of the device part correspond to the embodiments of the above-mentioned method, please refer to the above-mentioned method embodiments for the introduction of the elastic scheduling device provided by the present invention. The present invention will not be repeated here, and it has the same beneficial effects as the above-mentioned elastic scheduling method.
[0176] above Figure 3 The flexible scheduling device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The computer device in the embodiment of the present invention is described in detail from the perspective of hardware processing.
[0177] Figure 41 is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. The computer device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 510 (for example, one or more processors) and a memory 520, and one or more storage media 530 (for example, one or more mass storage devices) storing application programs 533 or data 532. Among them, the memory 520 and the storage medium 530 can be short-term storage or permanent storage. The program stored in the storage medium 530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the computer device 500. Furthermore, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the computer device 500.
[0178] The computer device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input and output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be appreciated by those skilled in the art that Figure 4 The illustrated computer device structure does not constitute a limitation on the computer device, and may include more or fewer components than illustrated, or combine certain components, or arrange the components differently.
[0179] The present invention also provides a computer device, which includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the elastic scheduling method in the above-mentioned embodiments.
[0180] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the flexible scheduling method.
[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.
[0183] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A flexible scheduling method, characterized in that: The flexible scheduling method includes: Monitor whether the task queues corresponding to each task node in the current cluster are empty; If there is an empty task queue, a capacity expansion plan corresponding to the elastic task node is formulated based on the preset capacity expansion strategy and cluster resource scheduling is performed; If there is a non-empty task queue currently, determine whether the cluster resources corresponding to the current non-empty task queue meet the current non-empty task queue; If the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, a scaling-down plan corresponding to the elastic task node is formulated based on the preset scaling-down strategy and cluster resource scheduling is performed.
2. The flexible scheduling method according to claim 1, characterized in that: If there is an empty task queue, then based on the preset expansion strategy, a capacity expansion plan corresponding to the elastic task node is formulated and cluster resource scheduling is performed, including: If there is an empty task queue, the trained prediction model is used to predict the idle resources of the task nodes corresponding to the empty task queue to obtain the idle resources in the current cluster. Based on the idle resources in the current cluster, the capacity expansion of elastic task nodes is calculated and multiple expansion plans are obtained; Score each expansion plan respectively, and use the expansion plan with the highest score as the expansion plan adopted by the elastic task node; Based on the final selected expansion plan, cluster resources are scheduled for the elastic task nodes to be expanded.
3. The flexible scheduling method according to claim 2, characterized in that: The prediction model includes a CPU prediction model, a memory prediction model and a GPU prediction model, and the prediction model is obtained by the following training method: Obtain monitoring data of different types of historical resources for each task node in the cluster; Select each target time point within the preset time length in turn, perform minute-level aggregation on the monitoring data of each target time point N days before each target time point, obtain a time series of the first length and use it as model input training data of the corresponding prediction model; Select each target time point within the preset time length in turn, perform minute-level aggregation on the monitoring data of each target time point M days after each target time point, obtain a time series of the second length and use it as model output training data of the corresponding prediction model; Based on various types of model input training data, model output training data and preset loss functions, different types of prediction models are trained respectively to obtain different types of prediction models trained by each task node in the cluster.
4. The flexible scheduling method according to claim 2, characterized in that: Based on the idle resources in the current cluster, the elastic task node is expanded and measured to obtain multiple expansion solutions including: Get the amount of resources used by the elastic task node for single-copy training; Taking the amount of resources used by the single replica as the expansion benchmark, based on the idle resources in the current cluster, the expansion of the elastic task node is measured and calculated to obtain multiple expansion plans; Each of the expansion plans includes the amount of resources used for the expansion and the expansion operation time, and the expansion operation time exceeds the task scheduling period of the elastic task node.
5. The flexible scheduling method according to claim 4, characterized in that: The method of scoring each expansion plan and using the expansion plan with the highest score as the expansion plan adopted by the elastic task node includes: Based on the amount of resources used for capacity expansion and the running time of capacity expansion in each of the capacity expansion plans, the score of each capacity expansion plan is calculated, and the capacity expansion plan with the highest score is used as the capacity expansion plan adopted by the elastic task node; The scoring formula is: S=F*T, where S represents the score value, F represents the amount of resources used for capacity expansion, and T represents the duration of capacity expansion operation.
6. The flexible scheduling method according to any one of claims 1 to 5, characterized in that: If there is a non-empty task queue currently, determining whether the current cluster resources satisfy the current non-empty task queue includes: If there is a non-empty task queue, the available resources of the cluster resources corresponding to the current non-empty task queue and the resources required by the tasks to be scheduled in the current non-empty queue are counted; If the available resource amount of the cluster resource corresponding to the current non-empty task queue is greater than or equal to the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource satisfies the current non-empty task queue; If the available resource amount of the cluster resource corresponding to the current non-empty task queue is less than the resource amount required by the task to be scheduled in the current non-empty queue, it is determined that the current cluster resource does not satisfy the current non-empty task queue.
7. The flexible scheduling method according to claim 6, characterized in that: If the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, a scaling-down plan corresponding to the elastic task node is formulated based on a preset scaling-down strategy and cluster resource scheduling is performed, including: If the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue, count whether the amount of resources available for scaling down the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue; If the amount of resources available for scaling down of the current elastic task node is greater than the amount of resources required by the tasks to be scheduled in the current non-empty queue, then based on the amount of resources required by the tasks to be scheduled in the current non-empty queue, the scaling down of the adapted elastic task node is calculated to obtain a scaling down plan; Based on the scaling-down solution, cluster resource scheduling is performed on the elastic task nodes to be scaled down.
8. A flexible scheduling device, characterized in that: The flexible scheduling device comprises: The monitoring module is used to monitor whether the task queue corresponding to each task node in the current cluster is empty; The capacity expansion module is used to formulate a capacity expansion plan corresponding to the elastic task node and perform cluster resource scheduling based on the preset capacity expansion strategy if there is an empty task queue at present; A judgment module is used to judge whether the cluster resources corresponding to the current non-empty task queue satisfy the current non-empty task queue if there is a non-empty task queue currently; The scaling-down module is used to formulate a scaling-down plan corresponding to the elastic task node and perform cluster resource scheduling based on a preset scaling-down strategy if the cluster resources corresponding to the current non-empty task queue do not satisfy the current non-empty task queue.
9. A computer device, characterized in that: The computer device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instruction in the memory to enable the computer device to execute the elastic scheduling method according to any one of claims 1 to 7.
10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the flexible scheduling method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Task scheduling method and device based on Jenkins and K8s and medium
CN121900921A