Method and apparatus for scheduling accelerator resources, electronic device, and medium

Abstract

Embodiments of the present disclosure relate to a method and apparatus for scheduling accelerator resources, an electronic device, and a medium. The method comprises acquiring an accelerator resource allocation request related to a machine learning model, wherein the accelerator resource allocation request indicates the number of accelerator resources required for a pod to run the machine learning model. The method further comprises, in response to the accelerator resource allocation request, triggering a rescheduling condition, and, on the basis of a rescheduling strategy, determining a pod to be rescheduled. In addition, the method further comprises, migrating the pod to be rescheduled from a first node to a second node, to allocate accelerator resources on the first node on the basis of the accelerator resource allocation request. According to the embodiments of the present disclosure, when the accelerator resource allocation request triggers the rescheduling condition, the pod to be rescheduled is determined on the basis of the rescheduling strategy, and then the pod to be rescheduled is migrated to another node to meet the resource allocation request, thereby improving the utilization efficiency of the accelerator resources.

Description

Method, device, electronic device, and medium for scheduling accelerator resources

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed on October 27, 2023, with application number 202311413945.1 and invention name “Method, device, electronic device and medium for scheduling accelerator resources”. The entire contents of that application are incorporated by reference into this application. Technical Field

[0003] The present disclosure relates generally to the field of computers, and more particularly, to a method, apparatus, electronic device, and medium for scheduling accelerator resources. Background Art

[0004] A machine learning model is one that learns from large amounts of data to predict new data. It automatically extracts useful features from the data and makes predictions based on these features. The learning process of a machine learning model typically consists of two phases: training and inference. During the training phase, the model learns and adjusts parameters based on known data. During the inference phase, the model uses data with known results to reason and evaluate the model's accuracy and generalization ability. Machine learning models have a wide range of applications, including but not limited to image recognition, speech recognition, natural language processing, and recommender systems. For example, in image recognition, machine learning models can automatically extract features from images by learning from large amounts of image data and predict the category of new images. In natural language processing, machine learning models can automatically extract linguistic features from text by learning from large amounts of text data and predict the topic of new text.

[0005] Machine learning models offer a wider range of applications and higher performance, but they also require longer training and inference times, larger models, and higher storage costs. Furthermore, machine learning models require higher computing power, more powerful computers, and more computing resources, resulting in higher costs.

[0006] Summary of the Invention

[0007] Embodiments of the present disclosure provide a method, apparatus, electronic device, and medium for scheduling accelerator resources.

[0008] According to a first aspect of the disclosure, a method for scheduling accelerator resources is provided. The method includes obtaining an accelerator resource allocation request related to a machine learning model, where the accelerator resource allocation request indicates the number of accelerator resources required for a container group running the machine learning model. The method also includes triggering a rescheduling condition in response to the accelerator resource allocation request, and determining a container group to be rescheduled according to a rescheduling strategy. In addition, the method also includes allocating accelerator resources for the accelerator resource allocation request on a first node by migrating the container group to be rescheduled from the first node to a second node.

[0009] In a second aspect of the disclosure, a device for scheduling accelerator resources is provided. The device includes a request acquisition module configured to acquire an accelerator resource allocation request related to a machine learning model, where the accelerator resource allocation request indicates the number of accelerator resources required for a container group running the machine learning model. The device also includes a container group determination module configured to trigger a rescheduling condition in response to the accelerator resource allocation request and determine a container group to be rescheduled according to a rescheduling strategy. In addition, the device also includes a container group migration module configured to allocate accelerator resources for the accelerator resource allocation request on a first node by migrating the container group to be rescheduled from the first node to the second node.

[0010] In a third aspect of the present disclosure, an electronic device is provided, comprising a processor and a memory coupled to the processor, wherein the memory has instructions stored therein, and when the instructions are executed by the processor, the electronic device executes the method according to the first aspect.

[0011] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to the first aspect.

[0012] This summary is intended to introduce a selection of concepts in a simplified form that are further described below in the detailed description. It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0014] FIG1 illustrates a schematic diagram of an example environment in which some embodiments of the present disclosure may be implemented;

[0015] FIG2 shows a flowchart of a method for scheduling accelerator resources according to some embodiments of the present disclosure;

[0016] FIG3 shows a schematic diagram of allocating accelerator resources according to some embodiments of the present disclosure;

[0017] FIG4 is a schematic diagram showing a workflow of a method for scheduling accelerator resources according to some embodiments of the present disclosure;

[0018] FIG5 shows a schematic diagram of the architecture of an example migration container group according to some embodiments of the present disclosure;

[0019] FIG6 is a schematic diagram showing the interaction between a scheduler and a daemon process in a container group according to some embodiments of the present disclosure;

[0020] FIG7 shows a block diagram of an apparatus for scheduling accelerator resources according to some embodiments of the present disclosure; and

[0021] FIG8 shows a block diagram of an example electronic device according to some embodiments of the present disclosure.

[0022] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0023] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, upon receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0029] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. can refer to different or the same objects, unless explicitly stated otherwise. Other explicit and implicit definitions may also be included below.

[0030] To enable more accelerator resources to participate in machine learning model development, various service models have emerged. Infrastructure as a Service (IaaS) provides information technology infrastructure as a service over the internet. Users can deploy and run arbitrary software, including operating systems and applications, for a fee. Platform as a Service (PaaS) provides a server platform as a service. In PaaS, users do not manage or control the underlying infrastructure, including networks, servers, operating systems, and storage. However, they can control deployed applications and possibly the configuration of the hosting environment in which they run. Within the service frameworks of PaaS and similar service models, more granular resource management methods, such as container group or container-level management, can be adopted. With container-level resource management, users only need to focus on the services within a container group or container, specifically their own data or model, training quality, and inference quality. In IaaS, once users have paid, the cloud service provider will not interfere with user resource usage, regardless of whether the infrastructure is in use or idle or loaded. In the Model as a Service (MaaS) framework, cloud service providers only benefit when users utilize the relevant resources. If there are a large amount of idle or fragmented computing resources on the platform, the cloud service provider's resource utilization will be reduced.

[0031] Various algorithms in traditional cloud service models fail to fully utilize accelerator resources, leading to frequent accelerator resource fragmentation. This phenomenon not only increases model training or inference latency but also wastes computing power, increasing operational costs. While one approach could ensure that accelerators in a cluster continue to meet computing power requirements by infinitely configuring accelerator resources, this approach does not address the issue of accelerator resource fragmentation. Furthermore, accelerator resources are expensive, adding additional costs. Another approach could address low accelerator resource utilization through accelerator resource virtualization, but accelerator resource virtualization cannot dynamically balance accelerator resource usage across the entire cluster. Yet another approach involves manually forcing accelerator resource selection, which incurs additional overhead. Therefore, to avoid the situation where a large number of idle, fragmented accelerator resources in a cluster can meet computing power requirements but remain underutilized, a method is needed to automatically address accelerator resource fragmentation in a cluster.

[0032] To this end, the embodiment of the present disclosure obtains an accelerator resource allocation request related to the machine learning model during the process of scheduling accelerator resources. The accelerator resource allocation request indicates the number of accelerator resources required for the container group running the machine learning model. If the accelerator allocation request can trigger the rescheduling condition, the container group to be rescheduled will be determined according to the rescheduling strategy. Then, the container group to be rescheduled is migrated from the original node to another node, so that accelerator resources are allocated to the container group running the machine learning model at the original node. Therefore, according to the embodiment of the present disclosure, idle fragmented accelerator resources can be automatically and fully utilized, which effectively speeds up the operation of machine learning model training or reasoning tasks and improves the utilization efficiency of accelerator resources.

[0033] FIG1 shows a schematic diagram of an example environment 100 in which some embodiments of the present disclosure may be implemented. In the example environment of the present disclosure, at least a plurality of accelerator clusters and a scheduler are included, wherein a plurality of accelerators are configured in the accelerator cluster. In some embodiments of the present disclosure, a graphics processing unit (GPU) is used as an example of an accelerator resource. However, embodiments of the present disclosure may also be used in combination with other accelerator resources. Referring to FIG1 , the example environment 100 includes at least a GPU cluster 110, a GPU cluster 120, a GPU cluster 130, and a scheduler 140.

[0034] As shown in Figure 1, GPU cluster 110 includes at least GPU 110-1, GPU 110-2, GPU 110-3, GPU 110-4, GPU 110-5, GPU 110-6, GPU 110-7, and GPU 110-8. It should be understood that the GPU cluster may include more or fewer GPUs. GPU cluster 120 includes at least GPU 120-1, GPU 120-2, GPU 120-3, GPU 120-4, GPU 120-5, GPU 120-6, GPU 120-7, and GPU 120-8. GPU cluster 130 includes at least GPU 130-1, GPU 130-2, GPU 130-3, GPU 130-4, GPU 130-5, GPU 130-6, GPU 130-7, and GPU 130-8. In some embodiments, the GPUs in GPU cluster 110, GPU cluster 120, or GPU cluster 130 are homogeneous, i.e., the GPUs are of the same type. Alternatively, the GPUs in GPU cluster 110, GPU cluster 120, or GPU cluster 130 are heterogeneous, i.e., the GPU cluster includes multiple GPUs of different types or from different manufacturers.

[0035] In some embodiments, GPU clusters 110, 120, and 130 are configured at different nodes, where a single container or container group can use one or more GPUs within the same node. In a GPU cluster, a container group is the smallest basic unit of deployment and management in the cluster, and a container group encapsulates one or more containers. Under normal circumstances, a container group will not be rescheduled after being bound to a node. The machine learning model file is loaded into the GPU cluster for operation. For example, a container can occupy 5 GPUs in the GPU cluster 110. These 5 GPUs may be GPU 110-1, GPU 110-2, GPU 110-3, GPU 110-4, and GPU 110-5, or they may be GPU 110-5, GPU 110-6, GPU 110-7, and GPU 110-8.

[0036] In some embodiments, when allocating GPU resources, the scheduler prioritizes allocating GPU resources across the entire machine without restarting the entire machine's accelerator resources. For example, if all GPUs in GPU clusters 110, 120, and 130 are idle, and a container group requests allocation of three GPU resources, the three idle GPUs in GPU cluster 110 can be allocated to that container group. Furthermore, if another container group requests allocation of one GPU resource, the scheduler will allocate one of the five idle GPUs in GPU cluster 110 to that container group, rather than allocating GPU resources from GPU clusters 120 and 130.

[0037] In some embodiments, the entire machine learning task is configured at the same node. For example, accelerator cluster 110 has 3 idle GPUs, accelerator cluster 120 has 2 idle GPUs, and accelerator cluster 130 has 1 idle GPU. When a machine learning task request is received and 4 GPU resources need to be allocated, the tasks of the container group of 3 GPU resources occupied in GPU cluster 130 are migrated to the 3 idle GPUs in GPU cluster 110, and then the machine learning task container group is configured in GPU cluster 130. This can avoid the delay caused by cross-node communication between GPUs and make full use of the fragmented GPUs in the GPU cluster. In some embodiments, the model file used for reasoning or training is scheduled by the scheduler to the GPU cluster for calculation or execution of other tasks.

[0038] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. The embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0039] The following describes in detail the process according to an embodiment of the present disclosure in conjunction with Figures 2 to 8. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0040] FIG2 shows a flowchart of a method 200 for scheduling accelerator resources according to some embodiments of the present disclosure, which may be performed by the scheduler 140 described in FIG1 . In block 202 , an accelerator resource allocation request related to a machine learning model is obtained, wherein the accelerator resource allocation request indicates the number of accelerator resources required by the container group running the machine learning model. For example, referring to FIG3 , the container group running the machine learning model requires 16 GPU resources, and a request 351 requests the accelerator resource set to allocate 16 GPU resources. In some embodiments, an accelerator resource allocation request related to a machine learning model may be obtained from a scheduler.

[0041] In block 204, in response to the accelerator resource allocation request triggering a rescheduling condition, the container group to be rescheduled is determined according to the rescheduling strategy. In some embodiments, whether the rescheduling condition has been triggered can be determined based on the relationship between the accelerator resource allocation request and the number of accelerator resources in the accelerator set. For example, referring to FIG3 , accelerator resource set 310, accelerator resource set 320, accelerator resource set 330, and accelerator resource set 340 are each configured with 8 accelerator resources, and there are currently 6 idle accelerator resources. After receiving the accelerator resource request 356 requesting the allocation of 4 GPU resources, it is determined whether the request 356 can trigger the rescheduling condition. For example, if the number of GPUs required by the request 356 is 4 less than the number of idle accelerator resources in the accelerator set 6 and the number of GPUs required by the request 356 is 4 greater than the number of remaining resources configured in each accelerator resource set, then it is confirmed that the request 356 has triggered the rescheduling condition.

[0042] In some embodiments, the container group to be rescheduled is determined based on the relationship between the accelerator resources required by the accelerator resource allocation request and the number of accelerator resources in the accelerator set. Continuing with FIG3 , if the number of idle accelerator resources in accelerator set 320 is 3, which is greater than or equal to the number of accelerator resources occupied by the container group to be scheduled in accelerator set 340, and the number of idle accelerator resources in accelerator set 340 after releasing 3 accelerator resources is 6, which is greater than or equal to the number of accelerator resources required by accelerator request 356, then the container group occupying 3 accelerator resources in accelerator resource set 340 is determined to be the container group to be scheduled, and the node where accelerator set 320 is located is determined to be the target node of the container group to be rescheduled.

[0043] In some embodiments, if there are multiple container groups to be rescheduled, the container group with the shortest time to reschedule can be selected as the container group to be rescheduled. For example, referring to FIG3 , when the accelerator resource allocation request 356 triggers the rescheduling condition, the container group occupying two accelerator resources in the accelerator set 340 can be selected as the container group to be rescheduled according to the scheduling policy, or the container group occupying three accelerator resources in the accelerator set 340 can be selected as the container group to be rescheduled. If the time it takes to reschedule the container group occupying two accelerator resources in the accelerator set 340 to the target node is shorter than the time it takes to reschedule the container group occupying three accelerator resources in the accelerator set 340 to the target node, then the container group occupying two accelerator resources in the accelerator set 340 is determined to be the container group to be rescheduled.

[0044] In some embodiments, if there are multiple container groups to be rescheduled, a container group that does not affect the system task process can be selected as the container group to be rescheduled. For example, the container group that performs training tasks can be selected as the container group to be rescheduled, and the container group that performs backend tasks can also be selected as the container group to be rescheduled, so as not to affect the normal supply of network services. In some embodiments, if multiple container groups to be rescheduled are all container groups that perform inference tasks, the container group with a lower priority is selected as the container group to be rescheduled. In some embodiments, the container group used to host network services or stateless services can be directly stopped, and then the container group can be quickly restarted at another node.

[0045] At block 206, accelerator resources are allocated to the accelerator resource allocation request on the first node by migrating the container group to be rescheduled from the first node to the second node. Referring to Figure 3 , accelerator resource request 356 requests allocation of four GPU resources. The container group in accelerator set 340, which occupies three GPU resources, is scheduled from the node where accelerator set 340 resides to the node where accelerator set 320 resides. GPU resources are then allocated to accelerator resource request 356 on the node where accelerator set 340 resides. The disclosed solution improves accelerator resource utilization.

[0046] In some embodiments, before migrating a container group to be rescheduled from its source node to its target node, it is necessary to stop processes within the rescheduled container group and save the state of related running data. For example, referring to FIG5 , when migrating container group 512 from host 570 to host 580, it is necessary to stop MaaS 514 within the container group and save its running data state, such as a checkpoint, to a shared file system 590.

[0047] In some embodiments, stopping a container group to be rescheduled involves interaction between the scheduler and the daemon process within the container group. For example, referring to Figure 6 , scheduler 620 sends a stop command to container group 616. Daemon process 617 within container group 616, upon receiving the command, saves the state file of the container group's running data to the file system and notifies the scheduler of the completion of the operation. In some embodiments, the state file of the running data is a checkpoint.

[0048] In some embodiments, after allocating resources to the container group to be rescheduled and allocating resources to the accelerator resource allocation request, the container group to be rescheduled and the container group running the machine learning model file can also be started at the same time. In some embodiments, the container group to be rescheduled and the container group running the machine learning model file have the same resource request priority, and this priority is higher than the resource request of any other container group. According to the solution disclosed in the present invention, the fragmented accelerator resources in the cluster can be effectively and fully utilized, the waiting time can be reduced, and the speed of the overall task process can be accelerated.

[0049] Figure 3 shows a schematic diagram of allocating accelerator resources 300 according to some embodiments of the present disclosure. As shown in Figure 3 , accelerator resource set 310 , accelerator resource set 320 , accelerator resource set 330 , and accelerator resource set 340 are each configured with eight accelerators.

[0050] Referring to Figure 3 , accelerator resource allocation request 351 requests the allocation of 16 accelerator resources. At this point, the accelerators in accelerator resource set 310, accelerator resource set 320, accelerator resource set 330, and accelerator resource set 340 are all idle. The idle accelerator resources in accelerator resource set 310 and accelerator resource set 320 can be allocated to request 351. After allocation 361, accelerator resource set 310 has 0 idle accelerators, accelerator resource set 320 has 0 idle accelerators, accelerator resource set 330 has 8 idle accelerators, and accelerator resource set 340 has 8 idle accelerators.

[0051] Next, the next accelerator resource allocation request 352 requests the allocation of six accelerator resources. At this point, accelerator resource set 310 has zero idle accelerators, accelerator resource set 320 has zero idle accelerators, accelerator resource set 330 has eight idle accelerators, and accelerator resource set 340 has eight idle accelerators. The idle accelerator resources in accelerator resource set 330 can be allocated to request 352. After allocation 362, accelerator resource set 310 has zero idle accelerators, accelerator resource set 320 has zero idle accelerators, accelerator resource set 330 has two idle accelerators, and accelerator resource set 340 has eight idle accelerators.

[0052] Continuing with Figure 3 , accelerator resource allocation request 353 requests the allocation of four accelerator resources. At this point, accelerator resource set 310 has zero idle accelerators, accelerator resource set 320 has zero idle accelerators, accelerator resource set 330 has two idle accelerators, and accelerator resource set 340 has eight idle accelerators. The idle accelerator resources in accelerator resource set 340 can be allocated to request 353. After allocation 363 , accelerator resource set 310 has zero idle accelerators, accelerator resource set 320 has zero idle accelerators, accelerator resource set 330 has two idle accelerators, and accelerator resource set 340 has four idle accelerators.

[0053] Continuing with Figure 3 , a new accelerator resource allocation request 354 requests the allocation of three accelerator resources. At this point, accelerator resource set 310 has zero idle accelerators, accelerator resource set 320 has zero idle accelerators, accelerator resource set 330 has two idle accelerators, and accelerator resource set 340 has four idle accelerators. The idle accelerator resources in accelerator resource set 340 can be allocated to request 354. After allocation 364, accelerator resource set 310 has zero idle accelerators, accelerator resource set 320 has zero idle accelerators, accelerator resource set 330 has two idle accelerators, and accelerator resource set 340 has one idle accelerator.

[0054] At this point, in the example of Figure 3 , accelerator resource release request 355 requests accelerator resource set 320 to release three accelerator resources. After release 365 , accelerator resource set 310 now has zero idle accelerators, accelerator resource set 320 has three idle accelerators, accelerator resource set 330 has two idle accelerators, and accelerator resource set 340 has one idle accelerator.

[0055] Next, a new accelerator resource allocation request 356 requests the allocation of four accelerator resources. At this point, accelerator resource set 310 has zero idle accelerators, accelerator resource set 320 has three idle accelerators, accelerator resource set 330 has two idle accelerators, and accelerator resource set 340 has one idle accelerator. Although there are six accelerator resources in the accelerator resource set, the idle accelerator resources are scattered across three different accelerator sets. To use these scattered accelerator resources, it is necessary to establish remote communication between these accelerators, such as through Transmission Control Protocol / Internet Protocol (TCP / IP) or Remote Direct Data Access (RDMA), or directly using communication channels between GPUs. This approach can reduce the performance of machine learning models because remote communication increases communication latency.

[0056] According to an embodiment of the present disclosure, a process previously assigned to three accelerator resources in accelerator resource set 340 can be scheduled to accelerator resource set 320, and the accelerators in accelerator resource set 340 can then be assigned to request 356. After rescheduling 366, accelerator resource set 310 has 0 idle accelerators, accelerator resource set 320 has 0 idle accelerators, accelerator resource set 330 has 2 idle accelerators, and accelerator resource set 340 has 4 idle accelerators. At this point, the four idle accelerators in accelerator resource set 340 are again assigned to request 356. After allocation 367, accelerator resource set 310 has 0 idle accelerators, accelerator resource set 320 has 0 idle accelerators, accelerator resource set 330 has 2 idle accelerators, and accelerator resource set 340 has 0 idle accelerators. In this way, the fragmented accelerator resources in the cluster can be fully utilized, thereby maximizing the utilization rate of accelerator resources without reducing the user experience of using accelerator resources.

[0057] In some embodiments, before scheduling a container group from accelerator resource set 320 to accelerator resource set 330, the running data state of the container group needs to be saved, such as by checkpointing it to the file system, and the container group needs to be stopped. In some embodiments, the container group to be rescheduled and the new container group have the same resource request priority, and this priority is higher than the resource request priority of other container groups except the container group to be rescheduled and the new container group.

[0058] Figure 4 illustrates a schematic diagram of a workflow diagram for a method 400 for scheduling accelerator resources according to some embodiments of the present disclosure. At block 402, the process for dynamically allocating accelerator resources is initiated. At block 404, an accelerator resource allocation request is requested, and an accelerator resource allocation request associated with a machine learning model file is obtained. This request indicates the number of accelerator resources required to run the machine learning model file. For example, referring to Figure 3, accelerator resource allocation request 351 requests the allocation of 16 GPU resources.

[0059] In block 406, a determination is made as to whether there are accelerator resources on the entire machine that satisfy the acquired accelerator resource allocation request. For example, if a request is made for 6 accelerator resources and there are 8 idle accelerators in the accelerator set 330 that can satisfy the request, the determination result is yes. For another example, if a request is made for 4 accelerator resources and there are only 3 idle accelerators in the accelerator set that can satisfy the request, the determination result is no.

[0060] If it is determined in block 406 that there are whole-machine accelerator resources that satisfy the acquired accelerator resource allocation request, then in block 410, the accelerator resources are directly allocated. For example, if a request is made for six accelerator resources, and there are eight idle accelerators in the accelerator set 330 that can satisfy the request, then six idle accelerators in the accelerator resource set 330 are allocated to the request. In block 416, the accelerator resources are used to process the requested task. For example, if the accelerator resource allocation request is for four accelerator resources, and there are eight idle accelerators in the accelerator set 330 that can satisfy the request, then in block 410, four idle accelerator resources in the accelerator set 330 are allocated to the request, and then in block 416, the allocated accelerator resources are used to process the task request. For another example, when a request is made to allocate three accelerator resources, there are four idle accelerators in the accelerator set 330 that can satisfy the request. Then, in box 410, four of the idle accelerator resources in the accelerator set 330 are allocated to the request, and then in box 416, the allocated accelerator resources are used to process the task request.

[0061] If, at block 406, it is determined that no whole-machine accelerator resources exist that satisfy the acquired accelerator resource allocation request, then at block 408, it is determined whether any fragmented accelerator resources exist that satisfy the request. If not, then at block 414, accelerator resources are added. If so, then at block 412, a container group to be rescheduled is searched. In some embodiments, at block 404, a request for accelerator resource allocation is received, for example, a new container group requests eight accelerator resources. Then, in box 406, it is determined whether there are whole-machine accelerator resources in the accelerator set that meet the request. For example, no whole-machine accelerator resources in accelerator set 310, accelerator set 320, accelerator set 330, and accelerator set 340 can meet the request. Then, in box 408, it is determined whether there are fragmented accelerator resources that meet the request. For example, if the remaining 6 accelerator resources in accelerator set 310, accelerator set 320, accelerator set 330, and accelerator set 340 cannot meet the request, and after searching a certain number of times (for example, 3 times), there are not enough fragmented accelerator resources released in this process to meet the request of the new container group. Then, in box 414, accelerator resources are added, for example, an accelerator cluster configured with 8 accelerator resources is added. Then, in box 410, the accelerator resources are allocated to the request, for example, the added 8 accelerator resources are allocated to the new container group request. Through this accelerator resource allocation method, the fragmented accelerator resources in the cluster can be fully utilized without having to wait for the release of other accelerator resources, thereby maximizing the utilization of accelerator resources.

[0062] In some embodiments, when request 356 requests allocation of four accelerator resources, the remaining six accelerator resources in accelerator set 310, accelerator set 320, accelerator set 330, and accelerator set 340 can satisfy the request. In block 412, a container group to be rescheduled is searched. For example, a container group occupying three accelerator resources in accelerator set 340 is found as the container group to be rescheduled. In block 418, the operation of the container group to be rescheduled is stopped. For example, stopping the operation of the container group occupying three accelerator resources in accelerator set 340 releases the accelerator resources in accelerator set 340 occupied by the container group, and saves the data state of the container group's operation to shared storage / file system. Subsequently, in block 420, the container group is scheduled to the target node, for example, to accelerator set 320. Then, in block 410, resources are allocated. For example, the idle resources in accelerator set 340 are allocated to the new container group, and the three idle accelerator resources in accelerator set 320 are allocated to the container group to be rescheduled, which previously released three accelerator resources in accelerator set 340. Next, in block 416, the container group is restarted from the saved data running state or checkpoint, and the new container group is started simultaneously. This accelerator resource allocation method can fully utilize fragmented accelerator resources in the cluster without having to wait for other accelerator resources to be released, thereby maximizing accelerator resource utilization.

[0063] In some embodiments, a request for allocation of accelerator resources from a new container group is obtained in block 404, for example, a request for allocation of four accelerator resources from the new container group. Subsequently, in block 406, it is determined whether there are any whole-machine accelerator resources in the accelerator set that meet the request. For example, if no whole-machine accelerator resources in accelerator set 310, accelerator set 320, accelerator set 330, or accelerator set 340 can meet the request, then in block 408, it is determined whether there are any fragmented accelerator resources that meet the request. For example, if there are a total of six remaining accelerator resources in accelerator set 310, accelerator set 320, accelerator set 330, and accelerator set 340 that can meet the request, then in block 412, a container group to be scheduled is searched. For example, a container group occupying one accelerator resource in accelerator set 330 is found as the container group to be scheduled. Then, in block 418, the operation of the container group to be rescheduled is stopped. For example, the operation of the container group occupying one accelerator resource in accelerator set 320 is stopped, thereby releasing the accelerator resource in accelerator set 330 occupied by the container group, and the data state of the container group's operation is saved to the shared storage / file system. Then, in block 420, the container group is scheduled to the target node, for example, to accelerator set 330. Subsequently, in block 410, resources are allocated, for example, allocating idle resources in accelerator set 320 to the new container group and allocating one idle accelerator resource in accelerator set 330 to the container group that previously released one accelerator resource in accelerator set 320. Next, in block 416, the container group is restarted at the saved data operation state and a new container group is simultaneously started to execute the task request.

[0064] In some embodiments, the container group to be scheduled is a container group that performs background tasks or training tasks. Background tasks are system-provided processes that can run in the background. Even if the application is suspended or no longer running, they do not affect the execution and provisioning of online services. In some embodiments, the container group to be scheduled has the same resource request priority as the new container group, and this priority is higher than the resource request priority of other container groups other than the container group to be scheduled and the new container group.

[0065] In some embodiments, the nearest deployment node can be selected as the target node of the container to be scheduled to achieve the nearest routing capability, thereby reducing the scheduling capability of the network loss. In some embodiments, when there are multiple container groups to be scheduled, the container group with the smallest scheduling delay is determined to be the container group to be scheduled. For example, if the time to schedule the container group to the node of the accelerator set 320 and restore the operation of the container group is longer than the time to schedule the container group to the node of the accelerator set 330 and restore the operation of the container group, the node of the accelerator set 330 is determined to be the target node of the container group to be scheduled. In this way, the latency impact of the container service can be reduced and the user service experience can be improved.

[0066] FIG5 shows a schematic diagram of the architecture 500 of an example migration container group of the present disclosure. As shown in FIG5 , host 570 and host 580 are both configured with multiple GPUs and some remote direct access network interface controllers (RNICs) at the hardware level. The RNICs can use direct memory access (DMA) when two or more computers are communicating, allowing direct access from the memory of one host to the memory of another host. The RNICs allow data to be transferred between services and storage settings. For example, host 580 configured with RNIC 584-1 and RNIC 584-2 can achieve direct memory access with host 570 also configured with RNIC 574-1 and RNIC 574-2, avoiding unnecessary delays. In some embodiments, the GPUs can be homogeneous or heterogeneous accelerators.

[0067] 5 , host 570 and host 580 respectively have a host operating system kernel 550 and a host operating system kernel 560 at the software level. Kernel 550 and 560 are configured with a kernel-based virtual machine (KVM) module 552 and a KVM module 562, respectively. KVM is a full virtualization solution based on the operating system kernel and employs hardware-assisted virtualization technology. In the KVM module, virtual machines are implemented as regular operating system processes and are scheduled by a standard operating system scheduler. Each virtual CPU (central processing unit) of a virtual machine is implemented as a regular operating system process, enabling KVM to utilize existing operating system kernel functionality.

[0068] Continuing with reference to Figure 5, an Elastic Compute Service (ECS) is configured on each host, such as ECS 510, ECS 520, ECS 530, and ECS 540. Each ECS contains one or more container groups, such as container group 512, container group 522, and container group 532. Each container group contains a MaaS, such as MaaS 514, MaaS 524, and MaaS 534. ECS is a resource collection consisting of CPU, memory, cloud disk, and so on. MaaS is a model that provides models as a service. It integrates model development, deployment, operation, and management into a unified platform, such as a container cluster platform, based on the cloud, and provides it to users, allowing them to easily use and manage models without having to worry about the model's implementation details and underlying technology.

[0069] In some embodiments, the ECS can be replaced with a bare metal server (BMS), which occupies the entire host (not shown). A BMS is a hardware device that combines the characteristics of a traditional physical server with the virtualization service capabilities of cloud computing technology, combining the advantages of hardware and software.

[0070] A container cluster platform is used to manage containerized applications across multiple hosts in a cloud platform. Within a container cluster platform, a pod (pod) is the smallest deployment and management unit within the cluster. A pod encapsulates one or more containers, storage resources, a separate network address, and policy options to manage and control how the containers operate. Each container is isolated from each other and has its own file system. A container contains the software packages for an application and its dependent environments, allowing applications to run in relatively isolated environments.

[0071] Using ECS ​​as the basic resource in the container service, you can build a highly available container cluster in the cloud with one click. In some embodiments, a node is an ECS, and users can flexibly choose the deployment method of the business according to business needs. In some embodiments, there is no restriction on the node type in the container service, and it can be a 32-bit operating system (X86), heterogeneous or bare metal type node. For example, the node configured in the host 570 is a bare metal type node, and the node configured in the host 580 is a cloud server type node. In some embodiments, a single container can use one or more accelerator resources of the same node, for example, the container of container group 512 can use GPU 572-1 on the host 570. For example, the container of container group 522 can use GPU 572-2 or GPU 572-3 on the host 570. For example, the container of container group 532 can use GPU 582-1 or GPU 582-2 or GPU 583-1 on the host 580, or any combination thereof.

[0072] Continuing to refer to Figure 5, the container group 512 and Maas 514 in the container group 512 are migrated from ECS 510 to ECS 540. Before the container group 512 is migrated, the status files of the data running in the container group need to be saved to the shared storage / file system 590. In some embodiments, the shared storage / file system 590 can be a distributed file system, including a local file system and a remote file system. In some embodiments, the container group to be scheduled is a container group that executes background tasks or training tasks. For example, the container group 512 is a container group that executes background tasks or a container group that executes training tasks. In this way, it is possible to avoid the scheduling behavior affecting the execution of foreground tasks and disrupting the normal program process. In some embodiments, if the container group 512 is a container group that executes a network hosting service or is in a no-service state, it can be shut down directly.

[0073] In some embodiments, a sidecar container is configured within the container group. A sidecar is a design pattern that separates application functionality from the application itself as a separate process, allowing for non-invasive additions to the application and avoiding the need to add extra code to meet third-party requirements. In the software architecture, a sidecar is attached to the main application, or parent application, to extend / enhance functionality, while the sidecar is loosely coupled to the main application. If a sidecar container provides an executable file that waits for the sidecar to be ready, the file can be called in the container's post-startup hook to prevent the startup of the remaining containers in the container group. Through the sidecar approach, the scheduler can interact with the daemon process within the container group.

[0074] In some embodiments, before scheduling container group 512 from ECS 510 to ECS 540, the scheduler sends a command to stop the operation of container group 512 to the daemon process in container group 512. After receiving the command, the daemon process in container group 512 stops the operation of container group 512 and saves the operation status of related data to shared storage / file system 590. Through this interactive operation, on the one hand, it effectively avoids the problem of the rescheduled container group starting from scratch and reducing the user experience, and on the other hand, it can fully utilize the fragmented accelerator resources in the cluster.

[0075] Figure 6 shows a schematic diagram 600 of the interaction between the scheduler and daemon processes within a container group, according to some embodiments of the present disclosure. As shown in Figure 6, cloud server 610 is configured with container groups 612, 614, and 616. Each container group is configured with a daemon process, which runs in the background and automatically exits when the main thread ends. The daemon processes can interact with the scheduler to control the container groups. For example, scheduler 620 can interact with daemon process 617 in container group 616, daemon process 615 in container group 614, and daemon process 613 in container group 612. Scheduler 620 sends a stop command to notify the container group to stop. Upon receiving the command, the daemon process within the container group instructs the container group's running status data to be saved to a shared file system and stops the container group. It then sends a notification message to inform the scheduler that the above operations have been completed. The scheduler is the component responsible for application scheduling. It schedules containers to run on working nodes by configuring node or container group affinity, etc.

[0076] In some embodiments, the scheduler 620 sends a stop command to notify the container group 616 to stop running. After receiving the command, the daemon process 617 in the container group 616 saves the data of the running status of the container group 616 to the shared file system, stops the running of the container group 616, and then sends a notification message to inform the scheduler 620 that the above operations have been completed. In some embodiments, the scheduler 620 monitors the status information of all accelerator resources in the cluster, that is, whether the accelerator resources are in an idle state or occupied state, and can update the status information of the accelerator resources after the accelerator resources are released or occupied.

[0077] Figure 7 shows a block diagram of an apparatus 700 for scheduling accelerator resources in accordance with some embodiments of the present disclosure. As shown in Figure 7, the apparatus 700 includes a request acquisition module 702 configured to acquire an accelerator resource allocation request related to a machine learning model, wherein the accelerator resource allocation request indicates the number of accelerator resources required for a container group running the machine learning model. The apparatus 700 also includes a container group determination module 704 configured to trigger a rescheduling condition in response to the accelerator resource allocation request and determine a container group to be rescheduled according to a rescheduling strategy. The apparatus 700 also includes a container group migration module 706 that allocates accelerator resources for the accelerator resource allocation request on the first node by migrating the container group to be rescheduled from the first node to the second node.

[0078] FIG8 shows a block diagram of an electronic device 800 of some embodiments of the present disclosure, and device 800 may be the device or apparatus described in an embodiment of the present disclosure. As shown in FIG8 , device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded from a storage unit 816 into a random access memory (RAM) 803. CPU / GPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804. Although not shown in FIG8 , device 800 may also include a coprocessor.

[0079] Various components in device 800 are connected to I / O interface 805, including: an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0080] The various methods or processes described above may be performed by the CPU / GPU 801. For example, in some embodiments, the methods may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the CPU / GPU 801, one or more steps or actions in the methods or processes described above may be performed.

[0081] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0082] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0083] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0084] The computer program instructions for performing the disclosed operation can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or the object code written in any combination of one or more programming languages, programming languages ​​include object-oriented programming languages, and conventional procedural programming languages.Computer-readable program instructions can be performed completely on a user's computer, partially on a user's computer, performed as an independent software package, partly on a user's computer and partly on a remote computer, or performed completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network-including local area network (LAN) or wide area network (WAN), or can be connected to an external computer (such as utilizing an internet service provider to connect by the internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to carry out personalized customization electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLA), this electronic circuit can perform computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0085] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0086] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0087] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented by a special hardware-based system that performs the specified function or action, or can be implemented by a combination of special hardware and computer instructions.

[0088] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements to existing technologies, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

[0089] Some example implementations of the present disclosure are listed below.

[0090] Example 1. A method for scheduling accelerator resources, comprising:

[0091] Obtain an accelerator resource allocation request related to a machine learning model, where the accelerator resource allocation request indicates a quantity of accelerator resources required for a container group to run the machine learning model;

[0092] In response to the accelerator resource allocation request triggering a rescheduling condition, determining a container group to be rescheduled according to a rescheduling policy; and

[0093] The accelerator resource is allocated on the first node for the accelerator resource allocation request by migrating the container group to be rescheduled from the first node to the second node.

[0094] Example 2. The method of Example 1, wherein the rescheduling condition includes at least one of the following:

[0095] The number of accelerator resources required by the container group running the machine learning model is greater than the remaining amount of accelerator resources on each node; and

[0096] The number of accelerator resources required for the container group to run the machine learning model is less than or equal to the total remaining amount of accelerator resources of each node.

[0097] Example 3. The method of any one of Examples 1-2, wherein determining the container group to be scheduled according to the rescheduling policy comprises:

[0098] In response to the fact that the idle amount of accelerator resources of one or more nodes is greater than or equal to the amount of accelerator resources occupied by the container group to be rescheduled, and in response to the fact that the idle amount of accelerator resources of the one or more nodes after the amount of accelerator resources occupied by the container group to be rescheduled is released is greater than or equal to the number of accelerator resources required for the container group to run the machine learning model, the container group to be rescheduled and the first node are determined.

[0099] Example 4. According to the method of any one of Examples 1-3, determining multiple candidate container groups, the multiple container groups including a first container group and a second container group; and

[0100] In response to determining that the restart duration of the second container group is greater than the restart duration of the first container group, the first container group is determined to be the container group to be rescheduled.

[0101] Example 5. The method of any one of Examples 1-4, wherein the container group to be rescheduled is not a container group that executes a front-end task.

[0102] Example 6. The method of any one of Examples 1-5, further comprising:

[0103] In response to the amount of accelerator resources required by the container group running the machine learning model being greater than the remaining amount of accelerator resources on the one or more nodes, configuring one or more new nodes.

[0104] Example 7. The method of any one of Examples 1-6, wherein migrating the container group to be rescheduled from the first node to the second node comprises:

[0105] Obtaining a stop instruction for the container group to be rescheduled, where the stop instruction instructs the container group to be rescheduled to stop running;

[0106] In response to obtaining the stop instruction, saving the running status data of the container group to be rescheduled; and

[0107] In response to receiving a notification indicating that the running state data of the to-be-rescheduled container group has been saved to the file system, stopping the running of the to-be-rescheduled container group.

[0108] Example 8. The method of any one of Examples 1-7, further comprising:

[0109] Starting a container group that runs the machine learning model on the first node; and

[0110] Allocate accelerator resources to the container group to be rescheduled on the second node.

[0111] Example 9. A method according to any one of Examples 1-8, wherein allocating accelerator resources to the container group to be rescheduled on the second node is based on the priority of the accelerator resource allocation request of the container group to be rescheduled being higher than the priority of other container groups on the second node.

[0112] Example 10. The method of any one of Examples 1-9, further comprising:

[0113] In response to the number of accelerator resources required for the container group running the machine learning model being less than or equal to the remaining amount of accelerator resources on one or more nodes, allocating accelerator resources for the accelerator resource allocation request in a node among the one or more nodes.

[0114] Example 11. An apparatus for scheduling accelerator resources, comprising:

[0115] a request acquisition module configured to acquire an accelerator resource allocation request related to a machine learning model, wherein the accelerator resource allocation request indicates a quantity of accelerator resources required for a container group running the machine learning model;

[0116] a container group determination module, configured to trigger a rescheduling condition in response to the accelerator resource allocation request, and determine a container group to be rescheduled according to a rescheduling policy; and

[0117] The container group migration module is configured to allocate accelerator resources for the accelerator resource allocation request on the first node by migrating the container group to be rescheduled from the first node to the second node.

[0118] Example 12. The apparatus of Example 11, wherein the rescheduling condition comprises at least one of:

[0119] The number of accelerator resources required by the container group running the machine learning model is greater than the remaining amount of accelerator resources on each node; and

[0120] The number of accelerator resources required for the container group to run the machine learning model is less than or equal to the total remaining amount of accelerator resources of each node.

[0121] Example 13. The apparatus of any of Examples 11-12, wherein the container group determination module comprises:

[0122] The first node determination module is configured to determine the container group to be rescheduled and the first node in response to the idle accelerator resource amount of one or more nodes being greater than or equal to the accelerator resource amount occupied by the container group to be rescheduled, and in response to the idle accelerator resource amount of the one or more nodes being greater than or equal to the number of accelerator resources required by the container group to run the machine learning model after the accelerator resource amount occupied by the container group to be rescheduled is released.

[0123] Example 14. The apparatus of any of Examples 11-13, further comprising:

[0124] a candidate container group determining module configured to determine a plurality of candidate container groups, the plurality of container groups including a first container group and a second container group; and

[0125] The module for determining a container group to be rescheduled is configured to, in response to determining that a restart duration of the second container group is longer than a restart duration of the first container group, determine that the first container group is the container group to be rescheduled.

[0126] Example 15. The apparatus of any of Examples 11-14, wherein the container group to be rescheduled is not a container group that executes a front-end task.

[0127] Example 16. The apparatus of any of Examples 11-15, further comprising:

[0128] The node configuration module is configured to configure one or more new nodes in response to the number of accelerator resources required by the container group running the machine learning model being greater than the remaining amount of accelerator resources on the one or more nodes.

[0129] Example 17. The apparatus of any of Examples 11-16, wherein the container group migration module comprises:

[0130] an instruction acquisition module, configured to acquire a stop instruction for the container group to be rescheduled, wherein the stop instruction instructs the container group to be rescheduled to stop running;

[0131] a data saving module, configured to save the running status data of the container group to be rescheduled in response to obtaining the stop instruction; and

[0132] The operation stopping module is configured to stop the operation of the container group to be rescheduled in response to receiving a notification indicating that the operation status data of the container group to be rescheduled has been saved to the file system.

[0133] Example 18. The apparatus of Examples 11-17, further comprising:

[0134] A start-up and operation module is configured to start and run the container group of the machine learning model on the first node; and

[0135] The resource allocation module is configured to allocate accelerator resources to the container group to be rescheduled on the second node.

[0136] Example 19. An apparatus according to any one of Examples 11-18, wherein allocating accelerator resources to the container group to be rescheduled on the second node is based on a priority of the accelerator resource allocation request of the container group to be rescheduled being higher than a priority of other container groups on the second node.

[0137] Example 20. The apparatus of any of Examples 11-19, further comprising:

[0138] A resource allocation module is configured to allocate accelerator resources for the accelerator resource allocation request in a node among the one or more nodes in response to the number of accelerator resources required for the container group running the machine learning model being less than or equal to the remaining amount of accelerator resources on the one or more nodes.

[0139] Example 21. An electronic device comprising:

[0140] processor; and

[0141] A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising:

[0142] Obtain an accelerator resource allocation request related to a machine learning model, where the accelerator resource allocation request indicates a quantity of accelerator resources required for a container group to run the machine learning model;

[0143] In response to the accelerator resource allocation request triggering a rescheduling condition, determining a container group to be rescheduled according to a rescheduling policy; and

[0144] The accelerator resource is allocated on the first node for the accelerator resource allocation request by migrating the container group to be rescheduled from the first node to the second node.

[0145] Example 22. The electronic device of Example 21, wherein the rescheduling condition comprises at least one of the following:

[0146] The amount of accelerator resources required by the container group running the machine learning model is greater than the remaining accelerator resources on each node, and

[0147] The number of accelerator resources required by the container group for running the machine learning model is less than or equal to the total remaining amount of accelerator resources of each node.

[0148] Example 23. The electronic device of any of Examples 21-22, wherein determining a container group to be scheduled according to a rescheduling policy comprises:

[0149] In response to the amount of free accelerator resources of one or more nodes being greater than or equal to the amount of accelerator resources occupied by the container group to be rescheduled, and in response to the amount of free accelerator resources of the one or more nodes being greater than or equal to the amount of accelerator resources required by the container group to run the machine learning model after the amount of accelerator resources occupied by the container group to be rescheduled is released, the container group to be rescheduled and the first node are determined.

[0150] Example 24. The electronic device of any of Examples 21-23, wherein the actions further comprise:

[0151] determining a plurality of candidate container groups, the plurality of container groups including a first container group and a second container group; and

[0152] In response to determining that the restart duration of the second container group is greater than the restart duration of the first container group, the first container group is determined to be the container group to be rescheduled.

[0153] Example 25. The electronic device of any of Examples 21-24, wherein the container group to be rescheduled is not a container group that executes a front-end task.

[0154] Example 26. The electronic device of any of Examples 21-25, wherein the actions further comprise:

[0155] In response to the amount of accelerator resources required by the container group running the machine learning model being greater than the remaining amount of accelerator resources on the one or more nodes, configuring one or more new nodes.

[0156] Example 27. The electronic device of any of Examples 21-26, wherein migrating the to-be-rescheduled container group from the first node to the second node comprises:

[0157] Obtaining a stop instruction for the container group to be rescheduled, where the stop instruction instructs the container group to be rescheduled to stop running;

[0158] In response to obtaining the stop instruction, saving the running status data of the container group to be rescheduled; and

[0159] In response to receiving a notification indicating that the running state data of the to-be-rescheduled container group has been saved to the file system, stopping the running of the to-be-rescheduled container group.

[0160] Example 28. The electronic device of any of Examples 21-27, wherein the actions further comprise:

[0161] Starting a container group that runs the machine learning model on the first node; and

[0162] Allocate accelerator resources to the container group to be rescheduled on the second node.

[0163] Example 29. An electronic device according to any one of Examples 21-28, wherein allocating accelerator resources to the container group to be rescheduled on the second node is based on a priority of the accelerator resource allocation request of the container group to be rescheduled being higher than a priority of other container groups on the second node.

[0164] Example 30. The electronic device of any of Examples 21-29, wherein the actions further comprise:

[0165] In response to the number of accelerator resources required for the container group running the machine learning model being less than or equal to the remaining amount of accelerator resources on one or more nodes, allocating accelerator resources for the accelerator resource allocation request in a node among the one or more nodes.

[0166] Example 31. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 10.

[0167] Example 32. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method of any one of Examples 1 to 10.

[0168] Although the present disclosure has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for scheduling accelerator resources, comprising: Obtain an accelerator resource allocation request related to a machine learning model, wherein the accelerator resource allocation request indicates a quantity of accelerator resources required for a container group to run the machine learning model; In response to the accelerator resource allocation request triggering a rescheduling condition, determining a container group to be rescheduled according to a rescheduling strategy; as well as The accelerator resource is allocated for the accelerator resource allocation request on the first node by migrating the container group to be rescheduled from the first node to the second node.

2. The method according to claim 1, wherein the rescheduling condition comprises at least one of the following: The number of accelerator resources required for the container group running the machine learning model is greater than the remaining amount of accelerator resources on each node; and The number of accelerator resources required for the container group to run the machine learning model is less than or equal to the total remaining amount of accelerator resources of each node.

3. The method according to claim 1, wherein determining the container group to be scheduled according to the rescheduling strategy comprises: In response to the amount of accelerator resource idleness of one or more nodes being greater than or equal to the amount of accelerator resources occupied by the container group to be rescheduled, and in response to the amount of accelerator resource idleness of the one or more nodes after the amount of accelerator resources occupied by the container group to be rescheduled is released being greater than or equal to the number of accelerator resources required for the container group to run the machine learning model, the container group to be rescheduled and the first node are determined.

4. The method according to claim 3, further comprising: determining a plurality of candidate container groups, the plurality of container groups comprising a first container group and a second container group; as well as In response to determining that the restart duration of the second container group is greater than the restart duration of the first container group, the first container group is determined as the container group to be rescheduled.

5. The method according to claim 3, wherein the container group to be rescheduled is not the container group that executes the front-end task.

6. The method according to claim 3, further comprising: In response to the number of accelerator resources required for the container group running the machine learning model being greater than the remaining amount of accelerator resources on the one or more nodes, one or more new nodes are configured.

7. The method according to claim 1, wherein migrating the container group to be rescheduled from the first node to the second node comprises: Obtaining a stop instruction for the container group to be rescheduled, wherein the stop instruction instructs the container group to be rescheduled to stop running; In response to obtaining the stop instruction, saving the running status data of the container group to be rescheduled; as well as In response to receiving a notification indicating that the running state data of the container group to be rescheduled has been saved to the file system, stopping the running of the container group to be rescheduled.

8. The method according to claim 1, further comprising: Starting a container group that runs the machine learning model on the first node; as well as Allocate accelerator resources for the container group to be rescheduled on the second node.

9. The method according to claim 8, wherein the accelerator resource is allocated to the container group to be rescheduled on the second node based on the priority of the accelerator resource allocation request of the container group to be rescheduled being higher than the priority of other container groups on the second node.

10. The method according to claim 1, further comprising: In response to the number of accelerator resources required for the container group running the machine learning model being less than or equal to the remaining amount of accelerator resources on one or more nodes, allocating accelerator resources for the accelerator resource allocation request in a node among the one or more nodes.

11. A device for scheduling accelerator resources, comprising: A request acquisition module is configured to acquire an accelerator resource allocation request related to a machine learning model, wherein the accelerator resource allocation request indicates the number of accelerator resources required for a container group to run the machine learning model; A container group determination module, configured to trigger a rescheduling condition in response to the accelerator resource allocation request, and determine a container group to be rescheduled according to a rescheduling strategy; as well as The container group migration module is configured to allocate accelerator resources for the accelerator resource allocation request on the first node by migrating the container group to be rescheduled from the first node to the second node.

12. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 10.

Images