A cluster resource scheduling method, system, terminal, and storage medium

Abstract

This invention discloses a cluster resource scheduling method, system, terminal, and storage medium. Based on mainstream AI computing platforms, it innovatively designs a resource manager, task lifecycle manager, and scheduler. When the job to be processed is an elastic training job, the allocated computing resources for the elastic training job are determined by the predicted idle resources of the cluster within a first preset time period and the required computing resources of the elastic training job. Based on the real-time idle resources of the cluster and the allocated computing resources, a job node is allocated to the job, and the job node is invoked to execute the elastic training job within a second preset time period. Based on the predicted idle resources of the cluster in the next first preset time period, the allocated computing resources for the elastic training job are adjusted. Based on the real-time idle resources of the cluster and the adjusted allocated computing resources, a job node is reallocated to the job, and the job node is invoked to continue executing the elastic training job in the next second preset time period, effectively improving cluster utilization.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a cluster resource scheduling method, system, terminal, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence, a single computing unit (e.g., an AI computing card) is no longer sufficient to support model training and optimization. Therefore, deep learning models are deployed in clusters to improve the efficiency of training and optimization by leveraging the computing resources provided by the cluster. A cluster consists of multiple nodes, and each node can be configured with multiple computing units.

[0003] The training and optimization of deep learning models can be divided into multiple tasks, each of which may require different amounts of computing resources. In existing technologies, the computing resources of the cluster are often evenly distributed or pre-configured for each task, resulting in low cluster utilization, severe task queuing, and low cluster efficiency. Summary of the Invention

[0004] The main objective of this invention is to provide a cluster resource scheduling method, system, terminal, and computer-readable storage medium, which aims to solve the problems of low cluster utilization, severe task queuing, and low cluster efficiency in the training and optimization of deep learning models in the prior art.

[0005] To achieve the above objectives, the present invention provides a cluster resource scheduling method, the method comprising:

[0006] Obtain pending jobs for training deep learning models;

[0007] When the job to be processed is an elastic training job, the allocated computing resources for the elastic training job are determined based on the predicted idle resources of the cluster in the first preset time period and the computing resources required by the elastic training job.

[0008] Based on the real-time idle resources of the cluster and the allocated computing resources of the elastic training job, nodes in the cluster are allocated to the elastic training job as job nodes, so that the job nodes can be invoked to execute the elastic training job within a second preset time period.

[0009] Based on the predicted amount of idle resources of the cluster in the next first preset time period, the amount of allocated computing resources for the elastic training job is adjusted.

[0010] Based on the real-time idle resource amount of the cluster and the adjusted allocated computing resource amount of the elastic training job, the job nodes of the elastic training job are reallocated to the elastic training job so that the reallocated job nodes can continue to execute the elastic training job in the next second preset time period, and continue to execute the step of adjusting the allocated computing resource amount of the elastic training job based on the predicted idle resource amount of the cluster in the next first preset time period, until the elastic training job is completed.

[0011] In some embodiments of the present invention, determining the allocated computing resources for the elastic training job based on the predicted idle resources of the cluster within a first preset time period and the required computing resources of the elastic training job specifically includes:

[0012] If the predicted idle resource amount in the first preset time period of the cluster is greater than the maximum value of the required computing resource amount of the elastic training job, the minimum value of the required computing resource amount of the elastic training job shall be used as the fixed resource amount of the elastic training job; and the first difference shall be used as the elastic resource amount of the elastic training job.

[0013] Wherein, the first difference is the difference between the maximum and minimum values ​​of the required computational resources for the elastic training task;

[0014] If the real-time idle resource quantity of the cluster is less than the maximum value of the required computing resource quantity of the elastic training job and is greater than or equal to the minimum value of the required computing resource quantity, the minimum value of the elastic training job shall be used as the fixed resource quantity of the elastic training job; and the second difference shall be used as the elastic resource quantity of the elastic training job.

[0015] Wherein, the second difference is the difference between the real-time idle resource amount of the cluster and the minimum value of the required computing resource amount of the elastic training job;

[0016] If the real-time idle resources of the cluster are less than the minimum value of the required computing resources of the elastic training job, the minimum value of the required computing resources of the elastic training job shall be used as the fixed resource amount of the elastic training job, and 0 shall be used as the elastic resource amount of the elastic training job.

[0017] The amount of computational resources allocated to the flexible training task is determined based on the fixed resource amount and the flexible resource amount of the task.

[0018] In some embodiments of the present invention, before determining the classified computing resource amount of the elastic training job based on the predicted idle resource amount of the cluster within a first preset time period and the resource amount calculated according to the demand of the elastic training job, the method further includes:

[0019] By using a preset cluster resource prediction model, the predicted available resource quantity of the cluster within the first preset time period is predicted; and

[0020] Obtain the amount of available resources released by the currently running pending jobs in the cluster during the first preset time period;

[0021] Based on the predicted available resources and the released available resources, the predicted idle resources of the cluster within the first preset time period are determined.

[0022] In some embodiments of the present invention, adjusting the allocated computing resources of the elastic training job based on the predicted idle resource amount of the cluster in the next first preset time period specifically includes:

[0023] If the predicted idle resource quantity of the cluster is greater than 0 and the job elastic resource quantity of the elastic training job is less than the first comparison value in the next first preset time period, the job elastic resource quantity of the elastic training job shall be expanded.

[0024] Wherein, the first comparison value is the difference between the maximum and minimum values ​​of the required computational resources for the elastic training task;

[0025] If the number of waiting job resources in the next first preset time period of the cluster is greater than 0 and the number of job elastic resources of the elastic training job is greater than 0, the number of job elastic resources of the elastic training job shall be reduced.

[0026] Based on the amount of elastic resources allocated to the elastic training job after scaling up or down, the amount of allocated computing resources for the elastic training job is adjusted.

[0027] In some embodiments of the present invention, when the predicted idle resource quantity of the cluster in the next first preset time period is greater than 0 and the job elastic resource quantity of the elastic training job is less than a first comparison value, the job elastic resource quantity of the elastic training job is expanded, specifically including:

[0028] If the predicted amount of idle resources in the cluster during the next first preset time period is greater than the second comparison value, the second comparison value will be used as the amount of expansion resources.

[0029] Wherein, the second comparison value is the difference between the maximum value of the required computational resources of the elastic training job minus the minimum value of the required computational resources and the elastic resource amount of the job.

[0030] If the predicted idle resource amount of the cluster in the next first preset time period is less than or equal to the second comparison value, the predicted idle resource amount of the cluster in the next first preset time period shall be used as the expansion resource amount.

[0031] Based on the increased resource capacity, the job elasticity resource capacity of the elastic training job is increased.

[0032] In some embodiments of the present invention, the step of allocating nodes of the cluster to the elastic training job as job nodes based on the real-time idle resource amount of the cluster and the allocated computing resource amount of the elastic training job specifically includes:

[0033] Determine whether the real-time idle resources of the cluster meet the allocated computing resources of the elastic training job;

[0034] If the real-time idle resources of the cluster meet the allocated computing resources of the elastic training job, a node of the cluster is allocated to the elastic training job as the job node of the elastic training job.

[0035] If the real-time idle resources of the cluster do not meet the allocated computing resources of the elastic training job, the elastic training job will be moved from the scheduling queue of the preset queue to the waiting scheduling queue of the preset queue.

[0036] After a preset waiting scheduling time, the elastic training job is readjusted from the waiting scheduling queue to the scheduling queue. If the real-time idle resources of the cluster meet the allocated computing resources of the elastic training job, a node of the cluster is allocated to the elastic training job as the job node until the elastic training job is executed.

[0037] In some embodiments of the present invention, after invoking the job node to execute the elastic training job within a second preset time period, the method further includes:

[0038] The elastic training job is moved from the scheduling queue of the preset queue to the secondary scheduling queue of the preset queue, so that after adjusting the allocated computing resources of the elastic training job based on the predicted idle resources of the cluster in the next first preset time period, the elastic training job is scheduled from the secondary scheduling queue, so that the elastic training job continues to execute in the next second preset time period.

[0039] In some embodiments of the present invention, after obtaining the job to be processed for training a deep learning model, the method further includes:

[0040] When the job to be processed is a flexible fixed job, the allocated computing resources for the flexible fixed job are determined based on the predicted idle resources of the cluster during the first preset time period and the required computing resources of the flexible fixed job.

[0041] Based on the real-time idle resources of the cluster and the allocated computing resources of the elastic fixed job, nodes of the cluster are allocated to the elastic fixed job as job nodes, so as to call the job nodes to execute the elastic fixed job.

[0042] In some embodiments of the present invention, after obtaining the job to be processed for training a deep learning model, the method further includes:

[0043] When the job to be processed is an inelastic job, the resource quantity is calculated based on the real-time idle resources of the cluster and the requirements of the inelastic job, and a node in the cluster is allocated to the inelastic job as the job node for the inelastic job, so as to call the job node to execute the inelastic job.

[0044] In some embodiments of the present invention, the method further includes:

[0045] If a preemptible job node is preempted by another job, the job being executed by the preempted job node is suspended. After the other job is completed, the preempted job node is controlled to continue executing the job.

[0046] To achieve the above objectives, the present invention also provides a cluster resource scheduling system, the system comprising: a job module, a resource manager, a task lifecycle manager, and a scheduler;

[0047] The job module is used to acquire jobs to be processed for training deep learning models;

[0048] The resource manager is used to determine the allocated computing resources for the elastic training job when the job to be processed is an elastic training job, based on the predicted idle resources of the cluster in a first preset time period and the resource requirements of the elastic training job.

[0049] The scheduler is used to allocate nodes in the cluster to the elastic training job as job nodes based on the real-time idle resources of the cluster and the allocated computing resources of the elastic training job, so as to call the job nodes to execute the elastic training job within a second preset time period.

[0050] The task lifecycle manager is used to adjust the allocated computing resources of the elastic training job based on the predicted amount of idle resources of the cluster in the next first preset time period.

[0051] The scheduler is further configured to reallocate the job nodes of the elastic training job according to the real-time idle resources of the cluster and the adjusted allocated computing resources of the elastic training job, so as to call the reallocated job nodes to continue executing the elastic training job in the next second preset time period, and continue to execute the step of adjusting the allocated computing resources of the elastic training job based on the predicted idle resources of the cluster in the next first preset time period, until the elastic training job is completed.

[0052] To achieve the above objectives, the present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the cluster resource scheduling method described in any of the above claims.

[0053] To achieve the above objectives, the present invention also provides a terminal, characterized in that it includes: a processor and a memory; the memory stores a computer-readable program that can be executed by the processor; when the processor executes the computer-readable program, it implements the steps in the cluster resource scheduling method described above.

[0054] This invention allocates computing resources to elastic training jobs by predicting the amount of idle resources in the cluster during a first preset time period. Then, based on the real-time idle resources and the allocated computing resources, it assigns corresponding job nodes to the elastic training job for execution. Furthermore, during the execution of the elastic training job, the allocated computing resources are adjusted according to the predicted idle resources in the next first preset time period. This involves reallocating job nodes to the elastic training job for execution during the next second preset time period. This dynamic adjustment of computing resources based on cluster resource changes fully utilizes fragmented idle resources, improves cluster resource utilization, alleviates severe cluster queuing issues, and accelerates the training of deep learning network models. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the structure of a cluster resource scheduling system provided in an embodiment of the present invention;

[0056] Figure 2 A flowchart of a cluster resource scheduling method provided in an embodiment of the present invention;

[0057] Figure 3 A flowchart of step S220 provided in an embodiment of the present invention;

[0058] Figure 4 A flowchart of step S310 provided in an embodiment of the present invention;

[0059] Figure 5 This is another flowchart of the cluster resource scheduling method provided in an embodiment of the present invention;

[0060] Figure 6 A flowchart of step S250 provided in an embodiment of the present invention;

[0061] Figure 7 A flowchart of step S610 provided in an embodiment of the present invention;

[0062] Figure 8 This is another flowchart of the cluster resource scheduling method provided in an embodiment of the present invention;

[0063] Figure 9 This is a schematic diagram of the terminal structure provided in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0065] Figure 1 This is a schematic diagram of the cluster resource scheduling system provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the cluster resource scheduling system provided in this embodiment of the invention mainly includes: job module 100, resource manager 200, task lifecycle manager 300, and scheduler 400.

[0066] The job module 100 is used to construct jobs to be processed for training deep learning models based on user operations, in order to obtain the jobs to be processed and their basic information.

[0067] The pending tasks include: flexible training tasks, flexible fixed tasks, and inflexible tasks. It can be understood that whether a task is a flexible training task, a flexible fixed task, or an inflexible task is determined by the user's actions; that is, it is specified by the user.

[0068] The basic information of a pending job can include job attributes and required computing resources. Job attributes include whether it can be preempted or not. Required computing resources refer to the amount of computing resources specified by the user for the pending job, which can be represented by the range [max, min]. In other words, the required computing resources for a pending job have a maximum value of max and a minimum value of min.

[0069] Preemptible means that a pending job can have its computing resources preempted by another pending job with higher priority during execution. The preemptible pending job is suspended and will resume execution using its original computing resources after the other higher-priority pending job has completed its execution. Non-preemptible means that if a pending job is preempted by another pending job with higher priority, an error will be reported, and it will not resume execution after the other higher-priority pending job has completed its execution.

[0070] It is understandable that the job attributes and required computing resources of the pending job are specified by the user when constructing the pending job, and the job module 100 can determine them through user operation.

[0071] Resource Manager 200 is used to predict the amount of idle resources in the cluster during the first preset time period, to determine the amount of computing resources allocated to elastic training jobs and elastic fixed jobs, and to monitor the amount of idle resources in the cluster in real time.

[0072] The allocated computing resources mentioned here refer to the computing resources allocated by the resource manager 200 to elastic training jobs and elastic fixed jobs based on their required computing resources. Furthermore, in this embodiment of the invention, non-elastic jobs do not require the resource manager 200 to allocate computing resources; that is, non-elastic jobs do not have allocated computing resources. In addition, the allocated computing resources for elastic training jobs are the sum of the job's elastic resources and fixed resources, while the allocated computing resources for elastic fixed jobs are simply the job's fixed resources.

[0073] The task lifecycle manager 300 is used to adjust the allocated computing resources for elastic training jobs based on the predicted idle resource amount of the cluster in different first preset time periods. In this embodiment of the invention, the task lifecycle manager 300 adjusts the allocated computing resources by adjusting the job elastic resource amount of the elastic training jobs.

[0074] The scheduler 400 allocates corresponding job nodes to pending jobs based on the real-time idle resource availability of the cluster obtained by the resource manager 200. The scheduler then executes the pending jobs by calling the corresponding job nodes, thereby training the deep learning model. The training mentioned here can refer to pre-training, model optimization, etc.

[0075] Therefore, in this embodiment of the invention, the elastic training job constructed by the user can dynamically adjust its computing resources according to the dynamic changes in cluster resources during execution. This is suitable for distributed machine learning, making full use of the available resources of the cluster for rapid model training, and is generally used for pre-training of deep learning models. Elastic fixed jobs, on the other hand, allocate corresponding computing resources based on the available resources of the cluster for training, and cannot be elastic based on dynamic changes in cluster resources. Non-elastic jobs do not require the allocation of computing resources based on the available resources of the cluster; they only require fixed computing resources specified by the user, and are commonly used for model performance improvement, ranking, and model parallelism in parallel computing.

[0076] In some embodiments of the present invention, the scheduler is provided with multiple preset queues, namely: a scheduling queue, a waiting scheduling queue, an unschedulable queue, a secondary scheduling queue, and a queue that can be forcibly occupied.

[0077] The scheduling queue contains jobs that are about to be scheduled. All newly created jobs are added to this queue to await scheduling.

[0078] The waiting queue contains pending jobs that failed to be scheduled initially due to cluster resource limitations. After a preset time, these pending jobs will be added back to the scheduling queue for scheduling.

[0079] The unschedulable queue contains pending jobs that failed to schedule due to issues such as crop parameter configuration or storage inability to bind. These jobs cannot be scheduled.

[0080] The secondary scheduling queue contains successfully scheduled elastic training jobs. These jobs await secondary scheduling, working in conjunction with the lifecycle controller to expand or reduce the allocated computing resources.

[0081] The queue that can be preempted is the super job. In this embodiment of the invention, if a super job fails to be scheduled during the initial scheduling due to insufficient idle resources in the cluster, it enters this queue. If, after a preset time, the idle resources in the cluster are still insufficient to meet the needs of the super job, then the super job can preempt the computing resources of any preemptible pending job.

[0082] The specific embodiments of the cluster resource scheduling system provided in this invention are basically the same as the specific embodiments of the cluster resource scheduling method described below, and will not be repeated here.

[0083] This invention provides a cluster resource scheduling method. Figure 2 A flowchart illustrating a cluster resource scheduling method provided in an embodiment of the present invention. Figure 2 As shown, the cluster resource scheduling method provided in this embodiment of the invention includes at least the following steps:

[0084] S210, Obtain the job to be processed for deep learning model training and the basic information of the job to be processed.

[0085] In this embodiment of the invention, the cluster resource scheduling method is applied to a cluster resource scheduling system.

[0086] Specifically, the job module 100 of the cluster resource scheduling system can obtain the jobs to be processed for deep learning model training and the basic information of the jobs to be processed based on user operations. As mentioned above, the jobs to be processed include at least three types: elastic training jobs, elastic fixed jobs, and non-elastic jobs; the basic information of the jobs to be processed can include at least: job attributes and required computing resources, which will not be elaborated here.

[0087] S220, when the job to be processed is an elastic training job, calculate the resource amount based on the predicted idle resource amount of the cluster in the first preset time period and the resource amount required by the elastic training job, and determine the allocated computing resource amount for the elastic training job.

[0088] Specifically, such as Figure 3 As shown, step S220 above can be achieved by at least the following methods:

[0089] S310: Obtain the predicted amount of idle resources in the cluster within the first preset time period.

[0090] In this embodiment of the invention, the system can be divided into multiple first preset time periods according to corresponding preset time intervals. The resource manager 200 obtains the predicted amount of idle resources within the first preset time periods.

[0091] Specifically, such as Figure 4 As shown, step S310 can be achieved through at least the following steps:

[0092] S410 uses a preset cluster resource prediction model to predict the amount of available resources for the cluster within a first preset time period.

[0093] In this embodiment of the invention, the preset cluster resource prediction model can be the ARIMA model.

[0094] Specifically, a preset cluster resource prediction model can be set in the resource manager 200. Taking the ARIMA model as an example, the ARIMA model set in the resource manager 200 for cluster resource prediction predicts the amount of available resources of the cluster within a certain time range in the future based on the cluster's historical resource information, that is, the predicted amount of available resources of the cluster within the first preset time period is obtained.

[0095] Resource Manager 200 can monitor the cluster's resource status in real time, meaning it can record historical resource information of the cluster at different time points at preset time intervals (e.g., minutes). The historical resource information for each time point includes: the total computing resources of the cluster at the current time point, and the amount of available resources (i.e., idle resources) of the cluster.

[0096] Furthermore, constructing an ARIMA model suitable for cluster resource prediction using historical data is existing technology and will not be elaborated upon in this embodiment of the invention.

[0097] The above scheme can predict the amount of idle resources in the cluster within a first preset time period based on the cluster's historical resource information, that is, predict the amount of idle resources.

[0098] S420: Obtain the amount of available resources released by currently executing pending jobs in the cluster within a first preset time period.

[0099] Specifically, the resource manager 200 can obtain the amount of available resources released by the currently executing pending jobs in the cluster within a first preset time period by using the task execution time of historical jobs.

[0100] In this embodiment of the invention, the resource manager 200 can first record and store the task execution time of completed jobs (i.e., historical jobs) and the basic information of historical jobs. Different versions of the same job have essentially the same task execution time, and different jobs with the same dataset and the same required amount of computing resources also have similar task execution times. Therefore, by obtaining the task execution time of the historical jobs corresponding to the currently executing jobs in the cluster, the amount of available resources released by the cluster within a first preset time period can be determined.

[0101] S430, based on the predicted available resources and released available resources of the cluster within the first preset time period, determine the predicted idle resources of the cluster within the first preset time period.

[0102] Specifically, Resource Manager 200 can predict the amount of available resources (i.e., predict the amount of idle resources) within a certain future timeframe using two dimensions: 1) the historical changes in the cluster's computing resources; and 2) the task execution times of historical jobs. The historical changes in the cluster's computing resources can be represented by the cluster's historical resource information.

[0103] Furthermore, the predicted available resources of the cluster during the first preset time period = a * predicted available resources + b * released available resources + z.

[0104] Wherein, a, b, and z are preset parameters that can be adjusted according to actual application conditions. In this embodiment of the invention, a = 0.8, b = 0.2, and z = 0 can be defaulted.

[0105] Through the above steps S410-S430, the idle resources of the cluster during the first preset time period can be fully predicted. The first preset time period can refer to 30 minutes, 1 hour, etc., and can be adjusted based on the actual situation.

[0106] S320, if the predicted idle resource amount of the cluster within the first preset time period is greater than the maximum value of the required computing resource amount of the elastic training job, the minimum value of the required computing resource amount of the elastic training job is used as the fixed resource amount of the elastic training job, and the first difference is used as the elastic resource amount of the elastic training job.

[0107] The first difference is the difference between the maximum value (max) and the minimum value (min) of the computational resources required for the elastic training task.

[0108] S330, if the predicted idle resource amount of the cluster within the first preset time period is less than the maximum value of the required computing resource amount of the elastic training job and is greater than or equal to the minimum value of the required computing resource amount, the minimum value of the elastic training job is used as the fixed resource amount of the elastic training job, and the second difference is used as the elastic resource amount of the elastic training job.

[0109] The second difference is the difference between the predicted amount of idle resources of the cluster during the first preset time period and the minimum value min of the computational resources required by the elastic training job.

[0110] S340, if the predicted idle resource amount of the cluster in the first preset time period is less than the minimum value of the required computing resource amount of the elastic training job, the minimum value of the elastic training job is used as the fixed resource amount of the elastic training job, and 0 is used as the elastic resource amount of the elastic training job.

[0111] As can be seen from the above embodiments, the fixed resource amount for an elastic training job is the minimum value of the computing resources allocated to the job, thus ensuring the basic resource requirements of the elastic training job. Furthermore, determining the elastic resource amount for the elastic training job based on the predicted idle resources of the cluster within a certain future timeframe allows for more efficient use of fragmented resources within the cluster, further improving cluster utilization and increasing the completion speed of the elastic training job.

[0112] S350, Based on the fixed resource quantity and flexible resource quantity of the flexible training task, determine the allocated computational resource quantity of the flexible training task.

[0113] The resource allocation for flexible training tasks is the sum of the fixed resource amount and the flexible resource amount for the task, i.e.:

[0114] The resource allocation for flexible training assignments is calculated as follows: Flexible resource allocation for assignments + Fixed resource allocation for assignments.

[0115] In this embodiment of the invention, the resource manager 200 can allocate corresponding computing resources to the elastic training job based on the available resources of the cluster within a certain time range in the future, thereby improving the cluster utilization, making full use of the resources in the cluster, and increasing the completion speed of the elastic training job.

[0116] S230 allocates nodes in the cluster to the elastic training job as job nodes based on the real-time idle resources of the cluster and the allocated computing resources of the elastic training job.

[0117] Specifically, the real-time idle resource quantity of the cluster can be obtained first; if the real-time idle resource quantity of the cluster is greater than or equal to the allocated computing resources of the elastic training job, a node in the cluster can be allocated to the elastic training job as the job node of the elastic training job.

[0118] Furthermore, the resource manager 200 obtains the real-time idle resource quantity of the cluster through real-time monitoring. The scheduler compares the real-time idle resource quantity of the cluster with the allocated computing resources of the elastic training job. If the real-time idle resource quantity of the cluster is greater than or equal to the allocated computing resources of the elastic training job, it indicates that the current cluster can execute the elastic training job. At this time, based on the resource usage of each node in the cluster, a corresponding node is allocated to the elastic training job as the job node.

[0119] It is understandable that the amount of computing resources available to each job node in the cluster may differ, as may the amount of idle resources available to each job node. Therefore, when a job node executes an elastic training job, the amount of computing resources utilized by each job node for that job may also differ.

[0120] For example, cluster A consists of multiple nodes, each node includes 8 AI computing cards, and the number of remaining available computing cards on each node is different. When the allocated computing resources for the elastic training job are 128, the elastic training job can be executed using 32 nodes, with each node allocated 4 AI computing cards.

[0121] As described above, the scheduler contains multiple preset queues, and the scheduling queue contains jobs to be scheduled. In some embodiments of the present invention, the jobs to be processed in the scheduling queue can be sorted, as described below:

[0122] Calculate the job score for each pending job based on its waiting time, allocated computing resources, priority, job attributes, and job type in the scheduling queue.

[0123] The jobs to be processed are sorted in the scheduling queue based on their job scores.

[0124] Furthermore, the score of the pending job = fx(waiting time + reciprocal of the allocated computing resources + priority + whether preemption is allowed + job type).

[0125] For example, the score of a pending job = a * waiting time + b * reciprocal of allocated computing resources + c * priority + d * whether preemption is allowed + e * job type. Here, a, b, c, d, and e are parameters, which can all be initialized to 1 and can be adjusted later according to usage.

[0126] In this embodiment of the invention, the higher the job score of the job to be processed, the closer it is to the front of the scheduling queue, and the earlier it can be scheduled.

[0127] The above-mentioned task types refer to whether the task to be processed is a flexible training task, a flexible fixed task, or a non-flexible task. Different scores can be pre-configured for different task types.

[0128] It is understandable that the above job attributes refer to whether or not preemption is allowed. Different scores can be pre-configured for jobs that can be preempted and those that cannot be preempted, so as to calculate the job score value of the job to be processed.

[0129] Using the above method, when there are many pending jobs in the scheduling queue, the jobs that are more urgently needed can be scheduled by sorting them.

[0130] Furthermore, in practical applications, there are situations where the real-time idle resources of the cluster are insufficient to meet the needs of elastic training jobs, such as... Figure 5 As shown, the cluster resource scheduling method provided in this embodiment of the invention may further include the following steps:

[0131] S510: If the real-time idle resources of the cluster do not meet the computing resources allocated to the elastic training job, the elastic training job will be added from the scheduling queue of the preset queue to the waiting scheduling queue.

[0132] The statement that the real-time idle resources of the cluster do not meet the allocated computing resources of the elastic training job means that the real-time idle resources of the cluster are less than the allocated computing resources of the elastic training job, resulting in the cluster being unable to execute the elastic training job at present.

[0133] Therefore, in this case, the scheduler moves the elastic training job from the scheduling queue to the waiting scheduling queue.

[0134] S520 adds the elastic training job from the waiting scheduling queue to the scheduling queue after a preset time, and allocates a node in the cluster to the elastic training job as the job node if the real-time idle resources of the cluster meet the allocated computing resources of the elastic training job, until the elastic training job is executed.

[0135] A waiting time can be set for pending jobs in the waiting queue of the scheduler. The scheduler will re-add elastic training jobs that have exceeded the waiting time to the scheduling queue to the scheduling queue. If the real-time idle resources of the cluster meet the computing resources allocated to the elastic training job, the scheduler will allocate a node of the cluster to the elastic training job as the job node of the elastic training job.

[0136] It is understandable that if the elastic training job that has been moved from the waiting scheduling queue to the scheduling queue still does not meet the required computing resources due to the real-time idle resources of the cluster, then step S610 above will continue to be executed until the elastic training job is executed.

[0137] S240, invoke the job node of the elastic training job and execute the elastic training job within the second preset time period.

[0138] In this embodiment of the application, after determining the job node to execute the elastic training job, the job node of the elastic training job is called and the elastic training job is executed within a second preset time period.

[0139] After assigning corresponding task nodes to the flexible training task, the task nodes for executing the flexible training task will remain unchanged within the second preset time period. The second preset time period mentioned here can be the same as or different from the first preset time period, and can be adjusted according to the actual situation.

[0140] S250 adjusts the classification computing resources of the elastic training job based on the predicted amount of idle resources in the cluster during the next first preset time period.

[0141] In this embodiment of the invention, the task lifecycle manager 300 adjusts the amount of computing resources allocated to the elastic training job based on the predicted amount of idle resources in the cluster during the next first preset time period.

[0142] Specifically, such as Figure 6 As shown, step S250 above can be achieved through at least the following steps:

[0143] S610, if the predicted idle resource quantity in the next first preset time period of the cluster is greater than 0 and the job elastic resource quantity of the elastic training job is less than the first comparison value, the job elastic resource quantity of the elastic training job is expanded.

[0144] The first comparison value is the difference between the maximum and minimum values ​​of the computational resources required for the flexible training task.

[0145] In this embodiment of the invention, the task lifecycle manager 300 monitors each elastic training job and adjusts the amount of computing resources allocated to the elastic training job.

[0146] Specifically, the task lifecycle manager 300 obtains the predicted amount of idle resources for the cluster in the next first preset time period through the resource manager 200. If the predicted amount of idle resources for the cluster in the next first preset time period is greater than 0, and the job elastic resource amount of the elastic training job is less than the first comparison value, the task lifecycle manager 300 expands the job elastic resource amount of the elastic training job.

[0147] Furthermore, such as Figure 7 As shown, step S610 above can be achieved through at least the following steps:

[0148] S710: If the predicted amount of idle resources in the cluster during the next first preset time period is greater than the second comparison value, the second difference will be used as the amount of resources to be expanded.

[0149] The second comparison value is the difference between the maximum and minimum values ​​of the required computational resources for the flexible training task, and the difference between the flexible resources for the task.

[0150] S720: If the predicted idle resource amount of the cluster in the next first preset time period is less than or equal to the second comparison value, the idle resource amount of the cluster in the next first preset time period shall be used as the expansion resource amount.

[0151] S730, based on the increased resource capacity, expands the elastic resource capacity of the elastic training job.

[0152] It is understandable that expanding the amount of flexible resources for flexible training tasks means adjusting the sum of the expanded resources and the amount of flexible resources for tasks to obtain the amount of flexible resources for tasks.

[0153] As can be seen from the above, the Task Lifecycle Manager 300 determines the amount of expansion resources for the elastic training job by predicting the amount of idle resources in the cluster during the next first preset time period.

[0154] S620 reduces the elastic resource quantity of the elastic training job when the waiting job resource quantity of the cluster is greater than 0 and the job elastic resource quantity of the elastic training job is greater than 0.

[0155] Among them, the amount of resources required for waiting jobs to be executed in the cluster resource scheduling and management system refers to the amount of resources required for jobs to be executed.

[0156] In this embodiment of the invention, the difference between the predicted idle resource amount in the previous first preset time period and the predicted idle resource amount in the next first preset time period can be used as the reduced resource amount, but it must be ensured that the reduced resource amount is greater than the minimum value of the resource range set for the elastic training task; then the job elastic resource amount of the elastic training job is reduced according to the reduced resource amount.

[0157] Similarly, it can be understood that reducing the amount of flexible resources for the flexible training task based on the reduced resource amount means using the difference between the current flexible resource amount and the reduced resource amount as the adjusted flexible resource amount.

[0158] S630: Adjust the allocated computing resources of the flexible training job based on the amount of flexible resources after the job has been expanded or reduced.

[0159] In this embodiment of the invention, the amount of elastic resources allocated to the elastic training job is expanded or reduced by predicting the amount of idle resources in the first preset time period of the cluster, thereby adjusting the amount of computing resources allocated to the elastic training job to adapt to the dynamic changes in cluster resources.

[0160] S260, based on the real-time idle resources of the cluster and the adjusted allocated computing resources of the elastic training job, reallocate job nodes for the elastic training job, so that the reallocated job nodes can continue to execute the elastic training job in the next second preset time period.

[0161] In this embodiment of the invention, the scheduler 400 reallocates job checkpoints for the elastic training job by obtaining the real-time idle resources of the cluster from the resource manager 200 and the adjusted allocated computing resources of the elastic training job, and calls the reallocated job nodes to continue executing the elastic training job in the next second preset time period.

[0162] It should be noted that step S260 can refer to step S230 above, and will not be repeated here.

[0163] S270, continue with step S240 until the elastic training job is completed.

[0164] The Task Lifecycle Manager 300 can monitor the progress of the elastic training job in real time. If the elastic training is not completed, it will continue to execute the above steps S250-S260 until the elastic training job is completed.

[0165] In some embodiments of the present invention, after the call job node executes the elastic training job in the second preset time period, the elastic training job can be added from the scheduling queue of the preset queue to the secondary scheduling queue, so that after adjusting the allocated computing resources of the elastic training job based on the predicted idle resources of the cluster in the next first preset time period, the elastic training job is scheduled from the secondary scheduling queue, so that the elastic training job continues to be executed in the next second preset time period.

[0166] In this embodiment of the invention, a secondary scheduling queue is provided for the flexible training job, thereby improving the accuracy of the flexible training job and effectively avoiding job omissions.

[0167] Furthermore, as can be seen from the above, the tasks to be processed can also be flexible fixed tasks. Therefore, such as Figure 8 As shown, after step S210, the cluster resource scheduling method provided by the present invention may further include the following steps:

[0168] S810: When the job to be processed is a flexible fixed job, the resource quantity to be allocated for the flexible fixed job is determined based on the predicted idle resource quantity of the cluster in the first preset time period and the resource quantity required by the flexible fixed job.

[0169] The amount of computational resources allocated for flexible fixed jobs differs from that allocated for flexible training jobs. The amount of computational resources allocated for flexible fixed jobs only includes the fixed resources allocated to the job.

[0170] Specifically, if the predicted amount of idle resources in the first preset time period of the cluster is greater than the maximum amount of computing resources required by the elastic fixed job, the resource manager 200 uses the maximum amount of computing resources required by the elastic fixed job as the job fixed resource amount of the elastic fixed job, which is the allocated computing resource amount of the elastic fixed job.

[0171] If the predicted idle resource amount in the first preset time period of the cluster is less than the maximum value of the required computing resource amount of the elastic fixed job but greater than the minimum value of the required computing resource amount of the elastic fixed job, the resource manager 200 uses the predicted idle resource amount in the first preset time period of the cluster as the job fixed resource amount of the elastic fixed job, that is, the allocated computing resource amount of the elastic fixed job.

[0172] If the predicted amount of idle resources in the cluster during the first preset time period is less than the minimum amount of computational resources required by the elastic fixed job, the resource manager 200 will use the estimated amount of resources as the job fixed resource amount for the elastic fixed job, which is the allocated computational resource amount for the elastic fixed job.

[0173] Wherein, estimated resource quantity = number of available resources at the average future waiting time - number of waiting tasks with higher priority * a - number of waiting tasks with lower priority * b.

[0174] The parameters a and b mentioned above are variable parameters and can be adjusted according to the actual situation.

[0175] For example: If the number of available resources at the average future waiting time is 128 cards, the resource manager 200, through accessing the interface of the intelligent scheduler, queries the waiting queue. Among the tasks currently waiting for resource allocation, the number of waiting cards for tasks with higher priority than the current task is 64, and the number of waiting cards for tasks with lower priority is 32. Then, using the above formula (assuming a = 1.2 and b = 0.8), the estimated number of resources is 25.6, rounded up to 26. In addition, if 26 is less than the minimum number of resources (min) for elastic fixed tasks, the value min is used; similarly, if it is greater than the maximum number of resources (max) for elastic fixed tasks, the value max is used.

[0176] S820 allocates cluster nodes to the elastic fixed job based on the real-time idle resources of the cluster and the allocated computing resources of the elastic fixed job, so as to call the job nodes to execute the elastic fixed job.

[0177] It is understandable that cluster nodes are allocated to elastic fixed jobs, and the specific method for allocating cluster nodes to elastic training jobs can be referred to, which will not be elaborated here.

[0178] In some embodiments of the present invention, when a preemptible pending job node is preempted by another pending job, the pending job being executed by the preemptible job node is suspended. After the other pending jobs have finished executing, the preempted job node resumes execution of the pending job. This scheme ensures the continuous execution of preemptible pending jobs, eliminating the need to restart execution after being preempted, thus further conserving cluster resources.

[0179] The cluster resource scheduling method provided by this invention allocates computing resources to elastic training jobs by predicting the amount of idle resources in the cluster during a first preset time period. This yields the allocated computing resources for the elastic training job. Based on the real-time idle resources and the allocated computing resources, a corresponding job node is assigned to the elastic training job, and the job node is invoked to execute the elastic training job. Furthermore, during the execution of the elastic training job, the allocated computing resources are adjusted according to the predicted idle resources in the next first preset time period. This reallocates job nodes to the elastic training job, and the newly assigned job nodes are invoked to compute and execute the elastic training job during the next second preset time period. This dynamic adjustment of the computing resources for the elastic training job based on changes in cluster resources fully utilizes fragmented idle resources, improves cluster resource utilization, alleviates severe cluster queuing problems, and accelerates the training of deep learning network models.

[0180] like Figure 1 As shown, the cluster resource scheduling system provided by the present invention includes a job module 100, a resource manager 200, a task lifecycle manager 300, and a scheduler 400.

[0181] The job module 100 is used to acquire jobs to be processed for training deep learning models.

[0182] Resource Manager 200 is used to calculate the amount of resources allocated to the elastic training job based on the predicted amount of idle resources in the cluster during the first preset time period and the needs of the elastic training job when the job to be processed is an elastic training job.

[0183] The scheduler 400 is used to allocate nodes in the cluster to the elastic training job based on the real-time idle resources of the cluster and the allocated computing resources of the elastic training job, so as to call the job node to execute the elastic training job within a second preset time period.

[0184] The Task Lifecycle Manager 300 is used to adjust the amount of computing resources allocated to elastic training jobs based on the predicted amount of idle resources in the cluster during the next first preset time period.

[0185] The scheduler 400 is used to reallocate the job node of the elastic training job according to the real-time idle resources of the cluster and the adjusted allocated computing resources of the elastic training job, so as to call the reallocated job node to continue to execute the elastic training job in the next second preset time period, and to make the task lifecycle manager 300 continue to adjust the allocated computing resources of the elastic training job based on the predicted idle resources of the cluster in the next first preset time period, until the elastic training job is completed.

[0186] Based on the above-described cluster resource scheduling method, the present invention also provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the cluster resource scheduling method described in the above embodiments.

[0187] Based on the above-described cluster resource scheduling method, this invention also provides a terminal, such as... Figure 9 As shown, it includes at least one processor 80; a display screen 81; and a memory 82, and may also include a communications interface 83 and a bus 84. The processor 80, display screen 81, memory 82, and communications interface 83 can communicate with each other via the bus 84. The display screen 81 is configured to display a preset user guide interface in the initial setup mode. The communications interface 83 can transmit information. The processor 80 can call logical instructions in the memory 82 to execute the cluster resource scheduling method in the above embodiment.

[0188] Furthermore, the logic instructions in the aforementioned memory 82 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0189] The memory 82, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 80 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 82, thereby implementing the methods in the above embodiments.

[0190] The memory 82 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal. Furthermore, the memory 82 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks; it may also be a transient storage medium.

[0191] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system, terminal, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0192] The system, terminal, and storage medium provided in this application are one-to-one corresponding to the method. Therefore, the system, terminal, and storage medium also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system, terminal, and storage medium will not be repeated here.

[0193] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0194] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0195] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A cluster resource scheduling method, characterized in that, The method includes: Obtain pending jobs for training deep learning models; When the job to be processed is an elastic training job, the allocated computing resources for the elastic training job are determined based on the predicted idle resources of the cluster in the first preset time period and the computing resources required by the elastic training job. The step of determining the allocated computing resources for the elastic training job based on the predicted idle resources of the cluster within a first preset time period and the computing resources required by the elastic training job specifically includes: If the predicted idle resource amount in the first preset time period of the cluster is greater than the maximum value of the required computing resource amount of the elastic training job, the minimum value of the required computing resource amount of the elastic training job shall be used as the fixed resource amount of the elastic training job; and the first difference shall be used as the elastic resource amount of the elastic training job. Wherein, the first difference is the difference between the maximum and minimum values ​​of the required computational resources for the elastic training task; If the real-time idle resource quantity of the cluster is less than the maximum value of the required computing resource quantity of the elastic training job and is greater than or equal to the minimum value of the required computing resource quantity, the minimum value of the elastic training job shall be used as the fixed resource quantity of the elastic training job; and the second difference shall be used as the elastic resource quantity of the elastic training job. Wherein, the second difference is the difference between the real-time idle resource amount of the cluster and the minimum value of the required computing resource amount of the elastic training job; If the real-time idle resources of the cluster are less than the minimum value of the required computing resources of the elastic training job, the minimum value of the required computing resources of the elastic training job shall be used as the fixed resource amount of the elastic training job, and 0 shall be used as the elastic resource amount of the elastic training job. The amount of computing resources allocated to the flexible training task is determined based on the fixed resource amount and the flexible resource amount of the task. Based on the real-time idle resources of the cluster and the allocated computing resources of the elastic training job, nodes in the cluster are allocated to the elastic training job as job nodes, so that the job nodes can be invoked to execute the elastic training job within a second preset time period. Based on the predicted amount of idle resources of the cluster in the next first preset time period, the amount of allocated computing resources for the elastic training job is adjusted. Based on the real-time idle resource amount of the cluster and the adjusted allocated computing resource amount of the elastic training job, the job nodes of the elastic training job are reallocated to the elastic training job so that the reallocated job nodes can continue to execute the elastic training job in the next second preset time period, and continue to execute the step of adjusting the allocated computing resource amount of the elastic training job based on the predicted idle resource amount of the cluster in the next first preset time period, until the elastic training job is completed.

2. The cluster resource scheduling method according to claim 1, characterized in that, Before determining the classification computing resource amount of the elastic training job based on the predicted idle resource amount of the cluster within a first preset time period and the resource amount required by the elastic training job, the method further includes: By using a preset cluster resource prediction model, the predicted available resource quantity of the cluster within the first preset time period is predicted; and Obtain the amount of available resources released by the currently running pending jobs in the cluster during the first preset time period; Based on the predicted available resources and the released available resources, the predicted idle resources of the cluster within the first preset time period are determined.

3. The cluster resource scheduling method according to claim 1, characterized in that, The step of adjusting the allocated computing resources of the elastic training job based on the predicted idle resource amount of the cluster in the next first preset time period specifically includes: If the predicted idle resource quantity of the cluster is greater than 0 and the job elastic resource quantity of the elastic training job is less than the first comparison value in the next first preset time period, the job elastic resource quantity of the elastic training job shall be expanded. Wherein, the first comparison value is the difference between the maximum and minimum values ​​of the required computational resources for the elastic training task; If the number of waiting job resources in the next first preset time period of the cluster is greater than 0 and the number of job elastic resources of the elastic training job is greater than 0, the number of job elastic resources of the elastic training job shall be reduced. Based on the amount of elastic resources allocated to the elastic training job after scaling up or down, the amount of allocated computing resources for the elastic training job is adjusted.

4. The cluster resource scheduling method according to claim 3, characterized in that, If the predicted idle resource quantity of the cluster is greater than 0 and the job elastic resource quantity of the elastic training job is less than a first comparison value during the next first preset time period, the job elastic resource quantity of the elastic training job is expanded, specifically including: If the predicted amount of idle resources in the cluster during the next first preset time period is greater than the second comparison value, the second comparison value will be used as the amount of expansion resources. Wherein, the second comparison value is the difference between the maximum value of the required computational resources of the elastic training job minus the minimum value of the required computational resources and the elastic resource amount of the job. If the predicted idle resource amount of the cluster in the next first preset time period is less than or equal to the second comparison value, the predicted idle resource amount of the cluster in the next first preset time period shall be used as the expansion resource amount. Based on the increased resource capacity, the job elasticity resource capacity of the elastic training job is increased.

5. The cluster resource scheduling method according to claim 1, characterized in that, The step of allocating nodes from the cluster to the elastic training job as job nodes based on the real-time idle resources of the cluster and the allocated computing resources of the elastic training job specifically includes: Determine whether the real-time idle resources of the cluster meet the allocated computing resources of the elastic training job; If the real-time idle resources of the cluster meet the allocated computing resources of the elastic training job, a node of the cluster is allocated to the elastic training job as the job node of the elastic training job. If the real-time idle resources of the cluster do not meet the allocated computing resources of the elastic training job, the elastic training job will be moved from the scheduling queue of the preset queue to the waiting scheduling queue of the preset queue. After a preset waiting scheduling time, the elastic training job is readjusted from the waiting scheduling queue to the scheduling queue. If the real-time idle resources of the cluster meet the allocated computing resources of the elastic training job, a node of the cluster is allocated to the elastic training job as the job node until the elastic training job is executed.

6. The cluster resource scheduling method according to claim 1, characterized in that, After invoking the job node to execute the elastic training job within a second preset time period, the method further includes: The elastic training job is moved from the scheduling queue of the preset queue to the secondary scheduling queue of the preset queue, so that after adjusting the allocated computing resources of the elastic training job based on the predicted idle resources of the cluster in the next first preset time period, the elastic training job is scheduled from the secondary scheduling queue, so that the elastic training job continues to execute in the next second preset time period.

7. The cluster resource scheduling method according to claim 1, characterized in that, After obtaining the jobs to be processed for training the deep learning model, the method further includes: When the job to be processed is a flexible fixed job, the allocated computing resources for the flexible fixed job are determined based on the predicted idle resources of the cluster during the first preset time period and the required computing resources of the flexible fixed job. Based on the real-time idle resources of the cluster and the allocated computing resources of the elastic fixed job, nodes of the cluster are allocated to the elastic fixed job as job nodes, so as to call the job nodes to execute the elastic fixed job.

8. The cluster resource scheduling method according to claim 1, characterized in that, After obtaining the jobs to be processed for training the deep learning model, the method further includes: When the job to be processed is an inelastic job, the resource quantity is calculated based on the real-time idle resources of the cluster and the requirements of the inelastic job, and a node in the cluster is allocated to the inelastic job as the job node for the inelastic job, so as to call the job node to execute the inelastic job.

9. The cluster resource scheduling method according to claim 1, characterized in that, The method further includes: If a preemptible job node is preempted by another job, the job being executed by the preempted job node is suspended. After the other job is completed, the preempted job node is controlled to continue executing the job.

10. A cluster resource scheduling system, characterized in that, The cluster resource scheduling system is used to implement the cluster resource scheduling method according to any one of claims 1-9, and the system includes: a job module, a resource manager, a task lifecycle manager, and a scheduler; The job module is used to acquire jobs to be processed for training deep learning models; The resource manager is used to determine the allocated computing resources for the elastic training job when the job to be processed is an elastic training job, based on the predicted idle resources of the cluster in a first preset time period and the resource requirements of the elastic training job. The scheduler is used to allocate nodes in the cluster to the elastic training job as job nodes based on the real-time idle resources of the cluster and the allocated computing resources of the elastic training job, so as to call the job nodes to execute the elastic training job within a second preset time period. The task lifecycle manager is used to adjust the allocated computing resources of the elastic training job based on the predicted amount of idle resources of the cluster in the next first preset time period. The scheduler is further configured to reallocate the job nodes of the elastic training job according to the real-time idle resources of the cluster and the adjusted allocated computing resources of the elastic training job, so as to call the reallocated job nodes to continue executing the elastic training job in the next second preset time period, and continue to execute the step of adjusting the allocated computing resources of the elastic training job based on the predicted idle resources of the cluster in the next first preset time period, until the elastic training job is completed.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the cluster resource scheduling method as described in any one of claims 1-9.

12. A terminal, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; when the processor executes the computer-readable program, it implements the steps of the cluster resource scheduling method as described in any one of claims 1-9.

Images