Resource scheduling method and device, electronic equipment and storage medium

Abstract

The invention provides a resource scheduling method and device, electronic equipment and a storage medium, which are applied to a resource management platform, and the platform is used for managing a GPU resource pool undertaking a model training task. The GPU resource pools comprise a first GPU resource pool in which a specified training task exclusively enjoys the computing power and a second GPU resource pool in which other model training tasks share the computing power; obtaining a running time period of the model training task which is completely run in the first GPU resource pool; determining a preemptable time period corresponding to the first GPU resource pool based on the operation time period of the model training task which is operated completely; the preemptable time period is an idle time period of the first GPU resource pool; and if the current moment belongs to the preemptable time period corresponding to the first GPU resource pool, scheduling a to-be-trained task running based on the GPU resources in the second GPU resource pool to the first GPU resource pool. According to the invention, the resource utilization rate of the first GPU resource pool in which the specified training task exclusively enjoys the computing power can be improved.

Description

Technical Field

[0001] One or more embodiments of the present disclosure relate to the technical field of GPU resource pool, and in particular, to a resource scheduling method, device, electronic device, and storage medium. Background Art

[0002] GPU Pooling is a resource management technology that allows multiple users or applications to share a set of GPU resources, rather than each user or application monopolizing one or more GPUs. This technology can improve GPU usage efficiency, reduce costs, and support more flexible resource allocation, facilitate centralized management and scheduling of multiple graphics processors (GPUs), and provide a flexible and efficient GPU computing resource service architecture.

[0003] In related technologies, for example, in public cloud services in a multi-tenant environment, different tenants may have their own GPU resource pools. In order to ensure data isolation and security, the usage status within a resource pool will not be disclosed to other resource pools to prevent resource competition between different tenants while protecting the privacy and security of each tenant's training tasks.

[0004] Among them, the GPU resource pool is divided into a GPU resource pool that exclusively uses computing power for designated model training tasks and a GPU resource pool that shares computing power with other model training tasks. Due to the isolation of computing power between GPU resource pools, they cannot perceive each other's usage status, resulting in the GPU resource pool that exclusively uses computing power for designated model training tasks being unable to schedule other queued training tasks during idle time, which will waste the idle time resources of the GPU resource pool that exclusively uses computing power for designated model training tasks, resulting in low R&D efficiency and a waste of manpower and material resources. Summary of the invention

[0005] The present disclosure provides a resource scheduling method, which is applied to a resource management platform; wherein the resource management platform is used to manage a GPU resource pool that undertakes a model training task; the GPU resource pool includes a first GPU resource pool whose computing power is exclusively shared by a designated model training task, and a second GPU resource pool whose computing power is shared by other model training tasks; the method includes:

[0006] Obtain the running time period of the model training task that has been completed in the first GPU resource pool;

[0007] Based on the running time period of the completed model training task, determine the preemptible time period corresponding to the first GPU resource pool; wherein the preemptible time period is an idle time period of the first GPU resource pool;

[0008] If the second GPU resource pool is running at full capacity, and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool, then the remaining time is calculated based on the current moment and the preemptible time period;

[0009] Schedule the target task to be trained in the second GPU resource pool to the first GPU resource pool, wherein the target task to be trained includes a task to be trained that is run based on the GPU resources in the second GPU resource pool, supports being scheduled to the first GPU resource pool, and is expected to run for no longer than the remaining duration.

[0010] Optionally, the method further includes:

[0011] Obtaining a task identifier of the task to be trained; wherein the task identifier includes a first type identifier for indicating that the task to be trained supports being scheduled to the first GPU resource pool; and a second type identifier for indicating that the task to be trained does not support being scheduled to the first GPU resource pool;

[0012] If the task identifier is the first type identifier, determining that the task to be trained supports being scheduled to the first GPU resource pool;

[0013] If the task identifier is the second type identifier, it is determined that the task to be trained does not support being scheduled to the first GPU resource pool.

[0014] Optionally, the method further includes:

[0015] Obtain the historical running time, data set size, and GPU computing resource specifications of each historical version task of the same type as the target task to be trained, as well as the data set size and GPU computing resource specifications of the target task to be trained;

[0016] According to the ratio of the data set size to the GPU computing resource specification of each historical version task, the historical running time of each historical version task is weighted and the average is calculated, and then multiplied by the ratio of the GPU computing resource specification to the data set size of the task to be trained to obtain the estimated running time of the target task to be trained.

[0017] Optionally, the method further includes:

[0018] If the target task to be trained that is scheduled to run in the first GPU resource pool has not been completed at the end of the preemptible period corresponding to the GPU resource pool, save the running information of the target task to be trained and remove the target task to be trained from the first GPU resource pool;

[0019] The operation information of the target task to be trained includes at least one of the following:

[0020] The model information used in the training of the target task to be trained, the optimizer used in the training of the target task to be trained, the round information of the training of the target task to be trained, and the loss function value during the training process of the target task to be trained.

[0021] Optionally, after saving the running information of the target task to be trained and withdrawing the target task to be trained from the first GPU resource pool, the method further includes:

[0022] If the current moment belongs to the preemptible time period corresponding to the other first GPU resource pool, then based on the current moment and the preemptible time period corresponding to the other first GPU resource pool, calculate the remaining time corresponding to the other first GPU resource pool;

[0023] If there is a remaining duration corresponding to the other first GPU resource pool that is not less than the estimated running duration of the target task to be trained, the target task to be trained is scheduled to the first GPU resource pool.

[0024] Optionally, the scheduling the target to-be-trained task in the second GPU resource pool to the first GPU resource pool includes:

[0025] Querying the ownership department of the second GPU resource pool about the remaining account period for borrowing resources from the first GPU resource pool;

[0026] If the remaining account period is less than the remaining duration, a recharge reminder message is sent to the ownership department of the second GPU resource pool to enable it to recharge and update the remaining account period; and / or based on the credit value of the ownership department of the second GPU resource pool, the remaining account period is updated to the remaining account period after the corresponding overdraft limit is increased;

[0027] If the updated remaining account period is not less than the remaining duration, the target task to be trained in the second GPU resource pool is scheduled to the first GPU resource pool.

[0028] Optionally, the method further includes:

[0029] If the number of tasks to be trained in the second GPU resource pool exceeds a preset threshold, and an instruction is received from the authority department of the second GPU resource pool that the tasks to be trained can be downgraded, the first GPU resource pool is divided into a plurality of sub-first GPU resource pools;

[0030] The GPU resources in each sub-first GPU resource pool only support the task to be trained to run in a degraded running mode, and the running time of the degraded running mode is longer than that of the non-degraded running mode.

[0031] The present disclosure also provides a resource scheduling device, which is applied to a resource management platform; wherein the resource management platform is used to manage a GPU resource pool that undertakes a model training task; the GPU resource pool includes a first GPU resource pool whose computing power is exclusively shared by a designated training task, and a second GPU resource pool whose computing power is shared by other model training tasks; the device includes:

[0032] An acquisition unit, used to acquire a running time period of a model training task that has been completed in the first GPU resource pool;

[0033] A determination unit, configured to determine a preemptible time period corresponding to the first GPU resource pool based on the running time period of the completed model training task; wherein the preemptible time period is an idle time period of the first GPU resource pool;

[0034] a calculation unit, configured to calculate a remaining duration based on the current moment and the preemptible time period if the second GPU resource pool is running at full capacity and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool;

[0035] A scheduling unit is used to schedule a target task to be trained in the second GPU resource pool to the first GPU resource pool, wherein the target task to be trained includes a task to be trained that is run based on the GPU resources in the second GPU resource pool, supports being scheduled to the first GPU resource pool, and is expected to run for no longer than the remaining duration.

[0036] The present disclosure also provides an electronic device, comprising a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0037] The memory stores machine-readable instructions, and the processor executes the above method by calling the machine-readable instructions.

[0038] The present disclosure also provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the above method is implemented.

[0039] Through the embodiments of the present disclosure, first obtain the running time period of the model training task that has been completed in the first GPU resource pool; then determine the preemptible time period corresponding to the first GPU resource pool based on the running time period of the model training task that has been completed; wherein the preemptible time period is the idle time period of the first GPU resource pool; if the second GPU resource pool is running at full capacity, and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool, then calculate the remaining time based on the current moment and the preemptible time period; finally, schedule the target task to be trained in the second GPU resource pool to the first GPU resource pool, and the target task to be trained is based on the GPU resources in the second GPU resource pool. It supports being scheduled to the first GPU resource pool, and the estimated running time is not greater than the remaining time. Accordingly, the present disclosure can improve the training resource utilization rate of the first GPU resource pool that exclusively shares the computing power of the specified training task, and realize the utilization of the idle time period of the first GPU resource pool that exclusively shares the computing power, thereby accelerating R&D efficiency and saving manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a schematic diagram of a GPU resource pool running a model training task, shown in an exemplary embodiment;

[0042] Figure 2 is a schematic diagram showing a method of scheduling a model training task in a GPU resource pool according to an exemplary embodiment;

[0043] Figure 3 is a flow chart of a resource scheduling method shown in an exemplary embodiment;

[0044] Figure 4 is a hardware structure diagram of an electronic device shown in an exemplary embodiment;

[0045] Figure 5 It is a block diagram of a resource scheduling device shown in an exemplary embodiment. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the technical solutions in the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present disclosure.

[0047] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this disclosure. In some other embodiments, the steps included in the method may be more or less than those described in this disclosure. In addition, a single step described in this disclosure may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this disclosure may also be combined into a single step for description in other embodiments.

[0048] GPU Pooling is a resource management technology that allows multiple users or applications to share a set of GPU resources, rather than each user or application monopolizing one or more GPUs. This technology can improve GPU usage efficiency, reduce costs, and support more flexible resource allocation, facilitate centralized management and scheduling of multiple graphics processors (GPUs), and provide a flexible and efficient GPU computing resource service architecture.

[0049] In some embodiments, such as in a public cloud service in a multi-tenant environment, different tenants may have their own GPU resource pools. To ensure data isolation and security, the usage status within a resource pool will not be disclosed to other resource pools to prevent resource competition between different tenants while protecting the privacy and security of each tenant's training tasks.

[0050] Among them, the GPU resource pool is divided into a GPU resource pool that exclusively uses computing power for designated model training tasks and a GPU resource pool that shares computing power with other model training tasks. Due to the isolation of computing power between GPU resource pools, they cannot perceive each other's usage status, resulting in the GPU resource pool that exclusively uses computing power for designated model training tasks being unable to schedule other queued training tasks during idle time, which will waste the idle time resources of the GPU resource pool that exclusively uses computing power for designated model training tasks, resulting in low R&D efficiency and a waste of manpower and material resources.

[0051] In view of this, the present disclosure aims to propose a technical solution for utilizing the idle resources of an exclusive computing power GPU resource pool.

[0052] During implementation, first, the running time period of the model training task that has been completed in the first GPU resource pool is obtained; further, based on the running time period of the model training task that has been completed, the preemptible time period corresponding to the first GPU resource pool is determined; wherein the preemptible time period is the idle time period of the first GPU resource pool; further, if the second GPU resource pool is running at full capacity, and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool, the remaining time is calculated based on the current moment and the preemptible time period; finally, the target task to be trained in the second GPU resource pool is scheduled to the first GPU resource pool, and the target task to be trained is run based on the GPU resources in the second GPU resource pool, supports being scheduled to the first GPU resource pool, and the estimated running time is not greater than the remaining time.

[0053] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram showing a GPU resource pool running a model training task according to an exemplary embodiment. Figure 1As shown, in the artificial intelligence platform, in order to meet the R&D needs of multiple enterprises or multiple business teams of an enterprise, it is often divided into multiple GPU resource pools for different algorithm or model R&D scheduling. Generally, there are multiple first GPU resource pools with exclusive computing power for designated model training tasks, and a second GPU resource pool with shared computing power for other model training tasks. According to the algorithm or model of each department, it is determined whether the department uses the first GPU resource pool with exclusive computing power or the second GPU resource pool with shared computing power. In this embodiment, Department A and Department B have their own first GPU resource pool, while Department C uses the second GPU resource pool with shared computing power. The artificial intelligence platform is used to manage GPU resource pools, monitor the status of GPU resource pools, manage model training services, and schedule and manage training tasks. In the first GPU resource pool A with exclusive computing power for Department A, version 1.0 of Task A, version 2.0 of Task A, and version 3.0 of Task A are running, and there is an idle period between the running of each task. In the first GPU resource pool B with exclusive computing power for department B, version 1.0 of task B and version 2.0 of task B are running, and there is an idle period between the running of each task. The running time of the completed model training tasks in the first GPU resource pool A with exclusive computing power for department A is 0:00-10:00, 11:00-19:00, 21:00-0:00, and the running time of the completed model training tasks in the first GPU resource pool B with exclusive computing power for department B is 0:00-13:00, 18:00-0:00. According to the model training tasks that have been completed in the two first GPU resource pools, the idle time periods corresponding to the two first GPU resource pools are determined as the preemptible time periods of the two first GPU resource pools. For example, the preemptible time periods corresponding to the first GPU resource pool A with exclusive computing power of department A are 10:00-11:00 and 19:00-21:00, and the preemptible time periods corresponding to the first GPU resource pool B with exclusive computing power of department B are 13:00-18:00. In these time intervals, for example, at 10:00, which is the preemptible time period of the first GPU resource pool A with exclusive computing power of department A, the task to be trained (task C1.0 version) that is run based on the GPU resources in the second GPU resource pool can be scheduled to the first GPU resource pool A with exclusive computing power of department A.

[0054] See also Figure 2 , Figure 2 FIG. 1 is a schematic diagram showing a method of scheduling a model training task in a GPU resource pool according to an exemplary embodiment. Figure 2As shown, in the preemptible time periods of the two first GPU resource pools, the first GPU resource pool A for exclusive computing power of department A and the first GPU resource pool B for exclusive computing power of department B, for example, the preemptible time periods corresponding to the first GPU resource pool A for exclusive computing power of department A are 10:00-11:00, 19:00-21:00, and the preemptible time periods corresponding to the first GPU resource pool B for exclusive computing power of department B are 13:00-18:00. If the current moment belongs to the preemptible time period corresponding to one of the two first GPU resource pools, the tasks to be trained (including: task C1.0 version, task D1.0 version, task E1.0 version) queued in the second GPU resource pool with shared computing power and whose estimated running time is not greater than the remaining time between the current moment and the end of the preemptible time period are scheduled to the first GPU resource pool B for exclusive computing power of department B in the first GPU resource pool, thereby utilizing the idle resources of the first GPU resource pool.

[0055] It should be noted that in this embodiment, only the preemptible time period of the first GPU resource pool when the current time is 10:00 is described by way of example, and the preemptible time period of the first GPU resource pool at other times is not limited. For example, 11:00, 14:00, and 18:00 are all preemptible time periods of the first GPU resource pool, and any time period can be within the range of the preemptible time periods of the two first GPU resource pools, which will not be described in detail here.

[0056] It can be seen that in the technical solution disclosed in the present invention, the running time period of the model training task that has been completed in the first GPU resource pool is first obtained; then, based on the running time period of the model training task that has been completed, the preemptible time period corresponding to the first GPU resource pool is determined; wherein, the preemptible time period is the idle time period of the first GPU resource pool; if the second GPU resource pool is running at full capacity, and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool, the remaining time is calculated based on the current moment and the preemptible time period; finally, the target task to be trained in the second GPU resource pool is scheduled to the first GPU resource pool, and the target task to be trained is based on the GPU resources in the second GPU resource pool. The task supports being scheduled to the first GPU resource pool, and the estimated running time is not greater than the remaining time. Accordingly, the present invention can improve the training resource utilization rate of the first GPU resource pool that exclusively shares the computing power of the specified training task, and realize the utilization of the idle time period of the first GPU resource pool that exclusively shares the computing power, thereby accelerating R&D efficiency and saving manpower and material resources.

[0057] Below through specific embodiments, Figure 1 and Figure 2The application scenarios shown are used to describe the present disclosure.

[0058] See also Figure 3 , Figure 3 This is a flow chart of a resource scheduling method shown in an exemplary embodiment. The method may perform the following steps:

[0059] Step 302: Obtain the running time period of the model training task that has been completed in the first GPU resource pool.

[0060] For example, there are two first GPU resource pools, namely, the first GPU resource pool A with exclusive computing power for department A and the first GPU resource pool B with exclusive computing power for department B. Among them, the running time periods of the model training tasks that have been completed in the first GPU resource pool A with exclusive computing power for department A are 0:00-10:00, 11:00-19:00, and 21:00-0:00, and the running time periods of the model training tasks that have been completed in the first GPU resource pool B with exclusive computing power for department B are 0:00-13:00 and 18:00-0:00.

[0061] Among them, the first GPU resource pool is a GPU resource pool with exclusive computing power. The GPU resource pool with exclusive computing power is specially allocated to a certain type or category of tasks and is not shared with other tasks to ensure that these tasks can obtain stable and sufficient computing resources. This resource allocation strategy can ensure the execution efficiency and quality of tasks, especially in deep learning and machine learning model training that require a lot of computing resources. For example, in the first GPU resource pool A where department A has exclusive computing power, from 0:00 to 10:00, all GPU resources are only used for the model training tasks required by department A and are not occupied by any other tasks. This means that during this time period, the model training tasks required by department A can make full use of all GPU resources to obtain the highest performance computing support.

[0062] Step 304: Based on the running time period of the completed model training task, determine the preemptible time period corresponding to the first GPU resource pool; wherein the preemptible time period is the idle time period of the first GPU resource pool.

[0063] For example, based on the running time periods of the model training tasks that have been completed in the first GPU resource pool A with the exclusive computing power of department A being 0:00-10:00, 11:00-19:00, and 21:00-0:00, and the running time periods of the model training tasks that have been completed in the first GPU resource pool B with the exclusive computing power of department B being 0:00-13:00 and 18:00-0:00, the idle time periods corresponding to the two first GPU resource pools are determined as the preemptible time periods of the two first GPU resource pools. For example, the preemptible time periods corresponding to the first GPU resource pool A with the exclusive computing power of department A are 10:00-11:00 and 19:00-21:00, and the preemptible time periods corresponding to the first GPU resource pool B with the exclusive computing power of department B are 13:00-18:00.

[0064] The first GPU resource pool A, which is used exclusively by department A, is a set of GPU resources reserved for a specific type of task (such as driving-related model training). The first GPU resource pool B, which is used exclusively by department B, is another set of GPU resources reserved for a specific task required by department B (such as testing or optimization-related model training). The present disclosure does not limit which specific task the first GPU resource pool is used to run.

[0065] Step 306: If the second GPU resource pool is running at full capacity and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool, the remaining time is calculated based on the current moment and the preemptible time period.

[0066] For example, at 13:00, the second GPU resource pool is running at full capacity, and 13:00 is the preemptible period corresponding to the first GPU resource pool B, where department B has exclusive computing power, that is, 13:00-18:00, and the remaining time is the time between 13:00 and 18:00, which is 5 hours.

[0067] Among them, the preemptible period refers to the period of time in a day when the GPU resource pool is not fully occupied by any model training task. In other words, this is an idle period of the GPU resource pool, which can be preempted and used by other types of model training tasks. For example, the preemptible period of the first GPU resource pool A, which has exclusive computing power for department A, is 10:00-11:00 and 19:00-21:00. This is because during these two periods, the GPU resources of the first GPU resource pool A, which has exclusive computing power for department A, are not occupied by driving-related model training tasks, and can therefore be used by other tasks. The remaining duration is the remaining duration in the preemptible period that can be used for resource scheduling.

[0068] Step 308: Schedule the target task to be trained in the second GPU resource pool to the first GPU resource pool, wherein the target task to be trained includes a task to be trained that is run based on the GPU resources in the second GPU resource pool, supports being scheduled to the first GPU resource pool, and has an estimated running time that is not greater than the remaining time.

[0069] For example, at 13:00, the remaining time of the first GPU resource pool B, which has exclusive computing power for department B, is 5 hours. The target task to be trained (such as task C1.0 version) with an estimated running time of no more than 5 hours and a scheduling flag configured to support scheduling can be scheduled to the first GPU resource pool B, which has exclusive computing power for department B, based on the GPU resources in the second GPU resource pool.

[0070] Among them, the estimated running time refers to the approximate time required to complete a specific task to be trained, which is estimated based on historical data, model complexity, data scale and other factors. In this embodiment, the system calculates the estimated running time of the task to be trained based on the GPU resources in the second GPU resource pool to determine whether it is suitable to be scheduled to the first GPU resource pool. If the estimated running time of the specific task to be trained is not greater than the preemptible time period corresponding to the first GPU resource pool, it is considered suitable to be scheduled to the first GPU resource pool, because the specific task to be trained can be completed at one time after scheduling. In order to ensure the accuracy of the calculation of the preemptible time period, it is usually considered that the estimated running time calculated by the model training task will have an error of about 5%, so as to correct the preemptible time period. The present disclosure does not limit the specific calculation and specific correction method of the preemptible time period. Whether it supports being scheduled to the first GPU resource pool for operation can be judged by the flag bit of the task to be trained. The present disclosure does not limit the specific setting method.

[0071] Through the above method, according to user needs, while making the best use of the idle resources of the exclusive computing power GPU resource pool, it is also possible to realize personalized scheduling of tasks in the shared computing power GPU resource pool.

[0072] In one embodiment shown, the method also includes: obtaining the task identifier of the task to be trained; wherein the task identifier includes a first type identifier for indicating that the task to be trained supports being scheduled to the first GPU resource pool; and a second type identifier for indicating that the task to be trained does not support being scheduled to the first GPU resource pool; if the task identifier is the first type identifier, determining that the task to be trained supports being scheduled to the first GPU resource pool; if the task identifier is the second type identifier, determining that the task to be trained does not support being scheduled to the first GPU resource pool.

[0073] For example, the task identifier of the C1.0 version of the task is obtained. The task identifier may be 1, indicating that the task to be trained supports being scheduled to the first GPU resource pool; and the task identifier may be 0, indicating that the task to be trained does not support being scheduled to the first GPU resource pool. If the task identifier of the C1.0 version of the task is 1, it is determined that the C1.0 version of the task supports being scheduled to the first GPU resource pool. If the task identifier of the C1.0 version of the task is 0, it is determined that the C1.0 version of the task does not support being scheduled to the first GPU resource pool.

[0074] The flag of the task identification may be a number or a letter, and the flag bit may be set to different values ​​or letters and set to represent different meanings, which is not limited in the present disclosure.

[0075] In one embodiment shown, the method also includes: obtaining the historical running time, data set size and GPU computing resource specifications of each historical version task of the same type as the target task to be trained, as well as the data set size and GPU computing resource specifications of the target task to be trained; according to the ratio of the data set size to the GPU computing resource specification of each historical version task, weighting the historical running time of each historical version task and calculating the average value, and then multiplying it by the ratio of the GPU computing resource specification to the data set size of the target task to be trained to obtain the expected running time of the target task to be trained.

[0076] For example, the target task to be trained is version 1.2 of Task C, which is run based on the GPU resources in the second GPU resource pool, with a data set size of 100MB and a GPU computing resource specification of 0.1 A100 GPU card. The historical version tasks of the same task type as version 1.2 of Task C are version 1.0 of Task C and version 1.1 of Task C, where the historical running time of version 1.0 of Task C is 6.5 hours, the data set size is 100MB, and the GPU computing resource specification is 0.1 A100 GPU card, and the historical running time of version 1.1 of Task C is 7h, the data set size is 100MB, and the resource specification is 0.1 A100 GPU card. Use the following formula to calculate the expected running time of version 1.2 of Task C:

[0077]

[0078] Among them, time_1.0 is the historical running time of the historical version 1.0 task with the same task type as the target task to be trained, time_1.1 is the historical running time of the historical version 1.1 task with the same task type as the target task to be trained, datasize_1.0 is the data set size of the historical version 1.0 task with the same task type as the target task to be trained, datasize_1.1 is the data set size of the historical version 1.1 task with the same task type as the target task to be trained, dataszie is the data set size of the target task to be trained, resources_1.0 is the GPU computing resource specification of the historical version 1.0 task with the same task type as the target task to be trained, resources_1.1 is the GPU computing resource specification of the historical version 1.1 task with the same task type as the target task to be trained, resource is the GPU computing resource specification of the target task to be trained, and the number of computing versions is the number of historical versions with the same task type as the target task to be trained.

[0079] In this embodiment, substituting the above formula, the estimated running time of version 1.2 of task C can be obtained:

[0080]

[0081] Among them, more GPU computing resource specifications can accelerate the processing speed of the task, otherwise it may extend the completion time of the target task to be trained. The larger the data set size of the target task to be trained, the more processing content of the target task to be trained, and the longer the processing time. The smaller the data set size of the target task to be trained, the less processing content of the target task to be trained and the shorter the processing time. Therefore, according to the ratio of the data set size of each historical version task to the GPU computing resource specification, the historical running time of each historical version task is weighted. Regarding the specific calculation method of the expected running time of the target task to be trained based on the GPU resources in the second GPU resource pool, the present disclosure does not limit this.

[0082] In the present disclosure, if there are idle resources in the second GPU resource pool, the target task to be trained is scheduled to run in the second GPU resource pool; if the current moment belongs to the preemptible time period corresponding to the first GPU resource pool, the target task to be trained that is run based on the GPU resources in the second GPU resource pool and has not been scheduled to the second GPU resource pool is scheduled to the first GPU resource pool to run the target task to be trained based on the GPU resources in the first GPU resource pool.

[0083] For example, Figure 1As shown, if there are idle resources in the second GPU resource pool of the shared computing power when preparing to run the target task to be trained (such as version 1.0 of task C), the target task to be trained is directly scheduled to run in the second GPU resource pool of the shared computing power without scheduling across GPU resource pools; if the current time is 13:00, which belongs to the preemptible time period corresponding to the first GPU resource pool, the target task to be trained that is run based on the GPU resources in the second GPU resource pool and has not been scheduled to the second GPU resource pool (such as version 1.0 of task D and version 1.0 of task E) is scheduled to the first GPU resource pool (the first GPU resource pool A with exclusive computing power of department A or the first GPU resource pool B with exclusive computing power of department B) to run the target task to be trained based on the GPU resources in the first GPU resource pool.

[0084] In this embodiment, if there are idle resources in the GPU resource pool of shared computing power, there is no need to schedule the target training tasks waiting to be run in the GPU resource pool of shared computing power across resource pools during the preemptible time period corresponding to the GPU resource pool of exclusive computing power, thereby saving computing power resources for task scheduling.

[0085] In one embodiment shown, the method also includes: if the target task to be trained scheduled to run in the first GPU resource pool has not been completed at the end of the preemptible time period corresponding to the GPU resource pool, then saving the running information of the target task to be trained and withdrawing the target task to be trained from the first GPU resource pool; wherein the running information of the target task to be trained includes at least one of the following: model information used for training the target task to be trained, optimizer used for training the target task to be trained, round information of training the target task to be trained, and loss function value during the training process of the target task to be trained.

[0086] For example, Figure 2As shown, the two first GPU resource pools are the first GPU resource pool A for exclusive computing power of department A and the first GPU resource pool B for exclusive computing power of department B. The preemptible time periods corresponding to the first GPU resource pool A for exclusive computing power of department A are 10:00-11:00 and 19:00-21:00, and the preemptible time periods corresponding to the first GPU resource pool B for exclusive computing power of department B are 13:00-18:00. In the preemptible time period of 10:00-11:00 corresponding to the first GPU resource pool A of the exclusive computing power of department A, the task to be trained (i.e., task C1.0 version) is scheduled to the first GPU resource pool A of the exclusive computing power of department A. When it reaches 11:00, it does not belong to the preemptible time period corresponding to the first GPU resource pool A of the exclusive computing power of department A. At this time, if task C1.0 version has not been completed, the running information of task C1.0 version is saved (including: the model information used for training of task C1.0 version (taking the image classification model trained by task C1.0 version as an example, the model information includes: model type ResNet-50, input size 224x224 RGB image, output category ImageNet dataset), the optimizer used for training of task C1.0 version is Adam, task C1.0 version has completed 5 rounds of training, and the loss function value of task C1.0 version during training is 0.23) and task C1.0 version is withdrawn from the first GPU resource pool A of the exclusive computing power of department A.

[0087] In this embodiment, it is considered that the estimated running time of the task to be trained may be inaccurate, or an emergency may occur during the actual operation, resulting in the task to be trained not being able to run completely in the preemptible period corresponding to the first GPU resource pool. At this time, the Checkpoint mechanism will be used, which is a mechanism in machine learning training, used to save the intermediate state of the model during the training process, including but not limited to model weights, optimizer status, training round information and loss function value. In this way, after the training is interrupted for various reasons (such as system restart, resource preemption, etc.), it can be restored from the last saved Checkpoint and continue training without starting from scratch, thereby saving time and computing resources. When the task to be trained is scheduled back to the GPU resource pool again, it can use the saved Checkpoint information to continue training from the interruption point without starting from scratch, which can effectively improve the flexibility and utilization of resources, while ensuring the continuity and efficiency of tasks. For the specific mechanism of breakpoint continuation, this disclosure does not limit this.

[0088] In one embodiment shown, after saving the running information of the target task to be trained and withdrawing the target task to be trained from the first GPU resource pool, the method also includes: if the current moment belongs to the preemptible time period corresponding to other first GPU resource pools, then based on the current moment and the preemptible time period corresponding to the other first GPU resource pools, calculating the remaining duration corresponding to the other first GPU resource pools; if there is a remaining duration corresponding to the other first GPU resource pool that is not less than the expected running duration of the target task to be trained, scheduling the target task to be trained to the first GPU resource pool.

[0089] For example, the two first GPU resource pools are the first GPU resource pool A for exclusive computing power of department A and the first GPU resource pool B for exclusive computing power of department B. The preemptible time periods corresponding to the first GPU resource pool A for exclusive computing power of department A are 10:00-13:00 and 19:00-21:00, and the preemptible time periods corresponding to the first GPU resource pool B for exclusive computing power of department B are 13:00-18:00. When the time reaches 13:00, it just does not belong to the preemptible time period corresponding to the first GPU resource pool A for exclusive computing power of department A. At this time, if the task C 1.0 version scheduled to run in the first GPU resource pool A for exclusive computing power of department A has not been completed, the running information of task C1.0 version is saved and task C1.0 version is withdrawn from the first GPU resource pool A for exclusive computing power of department A. Since 13:00 is the preemptible time period corresponding to the first GPU resource pool B of the exclusive computing power of department B, the remaining time is 5 hours, and the estimated running time of task C1.0 version is 3 hours, task C1.0 version is scheduled to run in the first GPU resource pool B of the exclusive computing power of department B.

[0090] It should be noted that if the running time or the remaining expected running time of task version C1.0 can be calculated based on the running information saved when task version C1.0 exits the first GPU resource pool A that exclusively enjoys computing power of department A, then when there are other first GPU resource pools with corresponding remaining times that are not less than the remaining expected running time of task version C1.0, task version C1.0 can be scheduled to other first GPU resource pools, which can increase the scheduling opportunities.

[0091] In one embodiment shown, the scheduling of the target task to be trained in the second GPU resource pool to the first GPU resource pool includes: querying the ownership department of the second GPU resource pool about the remaining account period for borrowing resources from the first GPU resource pool; if the remaining account period is less than the remaining duration, sending a recharge reminder message to the ownership department of the second GPU resource pool to enable it to recharge and update the remaining account period; and / or based on the credit value of the ownership department of the second GPU resource pool, updating the remaining account period to the remaining account period after increasing the corresponding overdraft limit; if the updated remaining account period is not less than the remaining duration, scheduling the target task to be trained in the second GPU resource pool to the first GPU resource pool.

[0092] For example, the department that owns the second GPU resource pool is Department C. The remaining account period for Department C to borrow resources from the first GPU resource pool is 4 hours. The preemptible time periods corresponding to the first GPU resource pool A, which has exclusive computing power for Department A, are 10:00-11:00 and 19:00-21:00. The preemptible time periods corresponding to the first GPU resource pool B, which has exclusive computing power for Department B, are 13:00-18:00. When it reaches 13:00, consider whether the training tasks to be run based on the GPU resources in the second GPU resource pool can be scheduled to the first GPU resource pool B, which has exclusive computing power for Department B. The remaining time for the first GPU resource pool B, which has exclusive computing power for Department B, is 5 hours between 13:00 and 18:00, and the remaining account period is 4 hours, which is less than the remaining time for the first GPU resource pool B, which has exclusive computing power for Department B. In this case, it is necessary to send a recharge reminder to department C to remind it to pay in order to continue the service of cross-GPU resource pool scheduling, or other processing methods, such as determining the corresponding overdraft limit (for example, 1 hour) based on department C's credit value or performance value in the previous year, and first granting department C an overdraft, so as to schedule the training task with an estimated running time of no more than 5 hours to run in the first GPU resource pool B with exclusive computing power for department B.

[0093] In this embodiment, it is ensured that each department can use resources fairly according to the payment situation within the specified time, and at the same time, through the setting of overdraft quota, additional support is provided to departments with good credit; in addition, different scheduling and overdraft strategies can be customized according to the needs and credit situation of different departments to increase the adaptability and personalization of the system. This not only improves the efficiency of resource use, but also increases the flexibility of the system and user satisfaction through intelligent management, and also provides an effective tool for enterprise resource management and cost control.

[0094] In one embodiment shown, the method also includes: if the number of tasks to be trained in the second GPU resource pool exceeds a preset threshold and an instruction is received from the authority department of the second GPU resource pool that allows the tasks to be trained to be downgraded, then the first GPU resource pool is divided into multiple sub-first GPU resource pools; wherein the GPU resources in each sub-first GPU resource pool only support the tasks to be trained to run in a downgraded mode, and the running time of the downgraded mode is higher than that of the non-degraded mode.

[0095] For example, an artificial intelligence research laboratory has two GPU resource pools (the first GPU resource pool, which is used exclusively by designated model training tasks, and the second GPU resource pool, which is used to share computing power with other model training tasks) to support machine learning model training tasks. Among them, the first GPU resource pool is allocated to specific advanced research projects that require a large amount of computing resources to train complex deep learning models. These projects are usually led by senior researchers who need to guarantee exclusive computing power to ensure the stability and efficiency of model training. The second GPU resource pool is used for model training tasks of other researchers, which may not have such high requirements for computing resources and can share resources for training. At a certain point in time, the number of tasks to be trained in the second GPU resource pool increased sharply to 12, exceeding the laboratory's preset threshold of 10 tasks to be trained. These additional tasks may be caused by the launch of new projects or the update of data sets. Due to the increase in the number of tasks, the computing resources of the second GPU resource pool become tight, which may affect the training efficiency of the tasks. In this case, the department that owns the second GPU resource pool in the laboratory's resource management team (for example, the log analysis department) instructs that the training task be allowed to run at a downgraded level, that is, to run under conditions where resources are less abundant, and to tolerate a slower computing speed to alleviate the resource shortage.

[0096] To achieve this goal, the first GPU resource pool is divided into multiple (for example, 4) sub-first GPU resource pools. Each sub-first GPU resource pool is configured to wait for receiving a task to be trained from the second GPU resource pool. In this way, some tasks to be trained that were originally queued in the second GPU resource pool can be scheduled to run in any of these sub-first GPU resource pools, thereby improving the overall resource utilization of the first GPU resource pool and the training efficiency of the tasks to be trained in the second GPU resource pool.

[0097] In this embodiment, the first GPU resource pool was originally used exclusively for a large deep learning project, but now it can be divided into four sub-GPU resource pools, each of which is assigned to a task to be trained. In this way, the four tasks to be trained can be run in the GPU resource pool that was originally used exclusively for one project, alleviating the pressure on the second GPU resource pool.

[0098] This flexible resource scheduling strategy enables the laboratory to manage GPU resources more effectively, ensuring that all research projects can complete model training within a reasonable time, while also improving resource utilization and the flexibility of research work.

[0099] It should be noted that, taking the example of dividing the first GPU resource pool into 4, the GPU computing resource specification of each sub-GPU resource pool after the division becomes 1 / 4 of the original. Therefore, the estimated running time of the task to be trained allocated to each sub-GPU resource pool after the division also becomes 4 times accordingly. The solution of this embodiment is suitable for tasks that do not require instant response and can accept a longer running time, such as log analysis, user behavior analysis, etc.

[0100] Among them, in the fields of machine learning and deep learning, there are some training tasks that have more flexible requirements for computing resources and can tolerate longer running times as long as they can start running as soon as possible. These tasks usually include: data preprocessing and cleaning tasks: These tasks usually involve the organization and preparation of large-scale data sets, such as normalization, denoising, missing value processing, etc. They do not require real-time processing and can run on slower GPU resources. Parameter tuning and model selection: Hyperparameter optimization and model selection processes usually require multiple iterations and evaluations. These tasks can tolerate slower computing speeds because they focus more on finding the optimal solution rather than speed. The present disclosure does not limit the types of tasks to be trained that are allocated to each sub-GPU resource pool after segmentation.

[0101] The common feature of these tasks is that they do not require real-time or near real-time response, so they can run in resource-constrained environments as long as execution can be started as early as possible. This flexibility allows these tasks to be trained to be scheduled between different computing resources to optimize overall resource utilization and task completion time. For example, if a large number of tasks to be trained need to be run in the second GPU resource pool, but the current second GPU resource pool does not have enough resources, the system can split the first GPU resource pool to schedule these tasks, even if this means that the running time will be longer, but the execution of the tasks can still be guaranteed. This scheduling strategy helps to improve resource utilization, reduce task waiting time, and ensure that research and development work can continue. Corresponding to the above-mentioned resource scheduling method embodiment, the present disclosure also provides an embodiment of a resource scheduling device.

[0102] See also Figure 4 , Figure 4 It is a hardware structure diagram of an electronic device shown in an exemplary embodiment. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present disclosure can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present disclosure do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0103] See also Figure 5 , Figure 5 5 is a block diagram of a resource scheduling device shown in an exemplary embodiment. The resource scheduling device 500 can be applied to Figure 4 In the electronic device shown, the technical solution of the present disclosure is implemented. The device is applied to a resource management platform; wherein the resource management platform is used to manage a GPU resource pool that undertakes model training tasks; the GPU resource pool includes a first GPU resource pool whose computing power is exclusively shared by the designated training task, and a second GPU resource pool whose computing power is shared by other model training tasks; the device includes:

[0104] An acquisition unit 502 is used to acquire a running time period of a model training task that has been completed in the first GPU resource pool;

[0105] A determination unit 504 is used to determine a preemptible time period corresponding to the first GPU resource pool based on the running time period of the completed model training task; wherein the preemptible time period is an idle time period of the first GPU resource pool;

[0106] A calculation unit 506 is configured to calculate a remaining duration based on the current moment and the preemptible time period if the second GPU resource pool is running at full capacity and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool;

[0107] The scheduling unit 508 is used to schedule the target task to be trained in the second GPU resource pool to the first GPU resource pool, wherein the target task to be trained includes a task to be trained that is run based on the GPU resources in the second GPU resource pool, supports being scheduled to the first GPU resource pool, and is expected to run for no longer than the remaining duration.

[0108] In this embodiment, the device further includes:

[0109] a second acquisition unit, configured to acquire a task identifier of the task to be trained; wherein the task identifier includes a first type identifier for indicating that the task to be trained supports being scheduled to the first GPU resource pool; and a second type identifier for indicating that the task to be trained does not support being scheduled to the first GPU resource pool;

[0110] A second determining unit is used to determine that the task to be trained supports being scheduled to the first GPU resource pool if the task identifier is the first type identifier;

[0111] The third determining unit is configured to determine that the task to be trained does not support being scheduled to the first GPU resource pool if the task identifier is the second type identifier.

[0112] In this embodiment, the device further includes:

[0113] A third acquisition unit is used to acquire the historical running time, data set size and GPU computing resource specifications of each historical version task of the same type as the target task to be trained, as well as the data set size and GPU computing resource specifications of the target task to be trained;

[0114] The second calculation unit is used to weight the historical running time of each historical version task according to the ratio of the data set size of each historical version task to the GPU computing resource specification, and then calculate the average value, and then multiply it by the ratio of the GPU computing resource specification to the data set size of the target task to be trained to obtain the estimated running time of the target task to be trained.

[0115] In this embodiment, the device further includes:

[0116] a saving unit, configured to save the running information of the target task to be trained and withdraw the target task to be trained from the first GPU resource pool if the target task to be trained that is scheduled to run in the first GPU resource pool has not been completed at the end of the preemptible period corresponding to the GPU resource pool;

[0117] The operation information of the target task to be trained includes at least one of the following:

[0118] The model information used in the training of the target task to be trained, the optimizer used in the training of the target task to be trained, the round information of the training of the target task to be trained, and the loss function value during the training process of the target task to be trained.

[0119] In this embodiment, after saving the running information of the target task to be trained and withdrawing the target task to be trained from the first GPU resource pool, the device further includes:

[0120] A third calculation unit is configured to calculate the remaining time corresponding to the other first GPU resource pool based on the current moment and the preemptible time corresponding to the other first GPU resource pool if the current moment belongs to the preemptible time period corresponding to the other first GPU resource pool;

[0121] The second scheduling unit is used to schedule the target task to be trained to the first GPU resource pool if there is another first GPU resource pool whose corresponding remaining duration is not less than the expected running duration of the target task to be trained.

[0122] In this embodiment, the scheduling unit includes:

[0123] A query subunit, used to query the ownership department of the second GPU resource pool for the remaining account period for borrowing resources from the first GPU resource pool;

[0124] a sending subunit, configured to send a recharge reminder message to the ownership department of the second GPU resource pool to enable it to recharge and update the remaining account period if the remaining account period is less than the remaining duration; and / or update the remaining account period to the remaining account period after adding the corresponding overdraft limit based on the credit value of the ownership department of the second GPU resource pool;

[0125] The scheduling subunit is used to schedule the training tasks based on the target GPU resources in the second GPU resource pool for operation if the updated remaining account period is not less than the remaining duration, and supports scheduling the training tasks to the first GPU resource pool whose running duration is expected to be no greater than the remaining duration to the first GPU resource pool.

[0126] In this embodiment, the device further includes:

[0127] The splitting unit is used to split the first GPU resource pool into multiple sub-first GPU resource pools if the number of tasks to be trained in the second GPU resource pool exceeds a preset threshold and an instruction is received from the authority department of the second GPU resource pool that the tasks to be trained are allowed to run in a downgraded manner; wherein the GPU resources in each sub-first GPU resource pool only support the tasks to be trained to run in a degraded operation mode, and the running time of the degraded operation mode is longer than that of the non-degraded operation mode.

[0128] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0129] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0130] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0131] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0132] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0133] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0134] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0135] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0136] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0137] The terms used in one or more embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present disclosure. The singular forms of "a", "said" and "the" used in one or more embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0138] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0139] The above description is only a preferred embodiment of one or more embodiments of the present disclosure and is not intended to limit one or more embodiments of the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present disclosure shall be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. A resource scheduling method, characterized in that: Applied to a resource management platform; wherein the resource management platform is used to manage a GPU resource pool that undertakes a model training task; the GPU resource pool includes a first GPU resource pool whose computing power is exclusively shared by a designated model training task, and a second GPU resource pool whose computing power is shared by other model training tasks; the method includes: Obtain the running time period of the model training task that has been completed in the first GPU resource pool; Based on the running time period of the completed model training task, determine the preemptible time period corresponding to the first GPU resource pool; wherein the preemptible time period is an idle time period of the first GPU resource pool; If the second GPU resource pool is running at full capacity, and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool, then the remaining time is calculated based on the current moment and the preemptible time period; Schedule the target task to be trained in the second GPU resource pool to the first GPU resource pool, wherein the target task to be trained includes a task to be trained that is run based on the GPU resources in the second GPU resource pool, supports being scheduled to the first GPU resource pool, and is expected to run for no longer than the remaining duration.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a task identifier of the task to be trained; wherein the task identifier includes a first type identifier for indicating that the task to be trained supports being scheduled to the first GPU resource pool; and a second type identifier for indicating that the task to be trained does not support being scheduled to the first GPU resource pool; If the task identifier is the first type identifier, determining that the task to be trained supports being scheduled to the first GPU resource pool; If the task identifier is the second type identifier, it is determined that the task to be trained does not support being scheduled to the first GPU resource pool.

3. The method according to claim 1, characterized in that The method further comprises: Obtain the historical running time, data set size, and GPU computing resource specifications of each historical version task of the same type as the target task to be trained, as well as the data set size and GPU computing resource specifications of the target task to be trained; According to the ratio of the data set size to the GPU computing resource specification of each historical version task, the historical running time of each historical version task is weighted and the average is calculated, and then multiplied by the ratio of the GPU computing resource specification to the data set size of the target task to be trained to obtain the estimated running time of the target task to be trained.

4. The method according to claim 1, characterized in that: The method further comprises: If the target task to be trained that is scheduled to run in the first GPU resource pool has not been completed at the end of the preemptible period corresponding to the GPU resource pool, save the running information of the target task to be trained and remove the target task to be trained from the first GPU resource pool; The operation information of the target task to be trained includes at least one of the following: The model information used in the training of the target task to be trained, the optimizer used in the training of the target task to be trained, the round information of the training of the target task to be trained, and the loss function value during the training process of the target task to be trained.

5. The method according to claim 4, characterized in that After saving the running information of the target task to be trained and withdrawing the target task to be trained from the first GPU resource pool, the method further includes: If the current moment belongs to the preemptible time period corresponding to the other first GPU resource pool, then based on the current moment and the preemptible time period corresponding to the other first GPU resource pool, calculate the remaining time corresponding to the other first GPU resource pool; If there is a remaining duration corresponding to the other first GPU resource pool that is not less than the estimated running duration of the target task to be trained, the target task to be trained is scheduled to the first GPU resource pool.

6. The method according to claim 1, characterized in that The scheduling of the target to-be-trained task in the second GPU resource pool to the first GPU resource pool includes: Querying the ownership department of the second GPU resource pool about the remaining account period for borrowing resources from the first GPU resource pool; If the remaining account period is less than the remaining duration, a recharge reminder message is sent to the ownership department of the second GPU resource pool to enable it to recharge and update the remaining account period; and / or based on the credit value of the ownership department of the second GPU resource pool, the remaining account period is updated to the remaining account period after the corresponding overdraft limit is increased; If the updated remaining account period is not less than the remaining duration, the target task to be trained in the second GPU resource pool is scheduled to the first GPU resource pool.

7. The method according to claim 1, characterized in that The method further comprises: If the number of tasks to be trained in the second GPU resource pool exceeds a preset threshold, and an instruction is received from the authority department of the second GPU resource pool that the tasks to be trained can be downgraded, the first GPU resource pool is divided into a plurality of sub-first GPU resource pools; The GPU resources in each sub-first GPU resource pool only support the task to be trained to run in a degraded running mode, and the running time of the degraded running mode is longer than that of the non-degraded running mode.

8. A resource scheduling device, characterized in that: Applied to a resource management platform; wherein the resource management platform is used to manage a GPU resource pool that undertakes a model training task; the GPU resource pool includes a first GPU resource pool whose computing power is exclusively shared by a designated training task, and a second GPU resource pool whose computing power is shared by other model training tasks; the device includes: An acquisition unit, used to acquire a running time period of a model training task that has been completed in the first GPU resource pool; A determination unit, configured to determine a preemptible time period corresponding to the first GPU resource pool based on the running time period of the completed model training task; wherein the preemptible time period is an idle time period of the first GPU resource pool; a calculation unit, configured to calculate a remaining duration based on the current moment and the preemptible time period if the second GPU resource pool is running at full capacity and the current moment belongs to the preemptible time period corresponding to the first GPU resource pool; A scheduling unit is used to schedule a target task to be trained in the second GPU resource pool to the first GPU resource pool, wherein the target task to be trained includes a task to be trained that is run based on the GPU resources in the second GPU resource pool, supports being scheduled to the first GPU resource pool, and is expected to run for no longer than the remaining duration.

9. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images