Managing accelerator resources of computer system

Abstract

Embodiments of the invention relate to managing accelerator resources of a computer system. In some implementations, a computer-implemented method includes monitoring usage of a first accelerator resource allocated to a first computing workload, and computing the first computing workload based on monitoring usage of the first accelerator resource allocated to the first computing workload. A determination is made that use of a first accelerator resource allocated to the first computing workload satisfies an idle condition. The method also includes reallocating the first accelerator resource to a second computing workload based on at least a determination that the use of the first accelerator resource allocated to the first computing workload satisfies an idle condition, the second computing workload being a pending computing workload.

Description

TECHNICAL FIELD

[0001] Embodiments of the application relate to managing accelerator resources of a computer system. BACKGROUND

[0002] Some computing environments can use one or more accelerators to more efficiently perform computing tasks. For example, a central processing unit (CPU) can offload or otherwise assign certain tasks to one or more accelerators for execution. As another example, a management computing node can assign processing tasks to accelerators in a cluster. Example accelerators can include graphics processing unit (GPU) devices, application specific integrated circuit (ASIC) devices, field programmable gate array (FPGA) devices, and visual processing unit (VPU) devices, and / or other types of devices. Although potentially used for a variety of purposes, computer systems can use these accelerators to accelerate the execution of compute-intensive algorithms, such as artificial intelligence processing, machine learning algorithms, or genomic sequence alignment algorithms. BRIEF DESCRIPTION OF DRAWINGS

[0003] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings in which:

[0004] Figure 1 illustrates an example computing system for managing accelerator resources according to certain implementations;

[0005] Figure 2 illustrates additional details of a scheduler node according to certain implementations;

[0006] Figure 3 illustrates an example scheduler according to certain implementations;

[0007] Figure 4 illustrates an example scheduler according to certain implementations;

[0008] Figure 5 illustrates additional details of an example workload queue of Figure 2 according to certain implementations;

[0009] Figure 6 illustrates an example workload information table that can include and / or be part of the workload information of Figure 1 according to certain implementations;

[0010] Figure 7 illustrates an example user interface according to certain implementations;

[0011] Figure 8 illustrates an example method for managing accelerator resources according to certain implementations;

[0012] Figure 9 FIGURE illustrates an example method for managing accelerator resources, in accordance with certain implementations;

[0013] Figure 10 FIGURE illustrates an example method for determining whether to reclaim an accelerator resource that has been allocated to a particular workload, in accordance with certain implementations; and

[0014] Figure 11 FIGURE illustrates a block diagram of an example computing device, in accordance with certain implementations. DETAILED DESCRIPTION

[0015] GPUs and other accelerators can be a needed computing resource that is often expensive and, therefore, can be scarce. Managing the use of accelerator resources presents certain challenges. In certain computing environments, a scheduler can be aware of computing workloads and can allocate accelerator resources for the computing workloads. For example, a scheduler of a containerized environment can receive computing workloads. These computing workloads can be JUPYTER NOTEBOOKS or another interactive application; however, the present disclosure contemplates any suitable type of computing workload that can be processed using a containerized environment or other computing environment. The scheduler can allocate one or more accelerator resources to a computing workload for execution of the computing workload. For example, the scheduler can allocate a particular GPU resource (e.g., one or more GPUs or one or more portions of one or more GPUs) to a computing workload for execution of the computing workload.

[0016] Whether in a containerized computing environment or another type of computing environment, allocating an accelerator resource to a computing workload can mean exclusive use of the accelerator resource by the computing workload for a particular period of time until the computing workload terminates and / or until another suitable event occurs. Thus, if a computing workload is in an idle state, the accelerator resource(s) allocated to the computing workload are also in an idle state. This is inefficient, particularly in situations where other computing workloads are in a pending state, waiting for an accelerator resource to become available, because the workload to which the accelerator resource is allocated allows the accelerator resource to sit idle, monopolizing the accelerator resource, while other computing workloads wait in a queue of pending workloads, resulting in underutilization of the accelerator resource.

[0017] Certain implementations of the present disclosure provide techniques for automatic resource reclamation for idle accelerator resources (e.g., GPUs). In certain implementations, a scheduler, which can be implemented as a standalone scheduler or as a plugin to another scheduler, can monitor usage of accelerator resources that are allocated to compute workloads. For example, the accelerator resources can report usage information and / or the workloads themselves can include certain information (e.g., priority information, utilization threshold, and / or any other suitable information). The scheduler can determine, based on monitoring usage of the accelerator resources, that usage of a particular accelerator resource satisfies an idle condition. The idle condition can be implemented as an idle threshold, and determining that usage of the particular accelerator resource satisfies the idle condition can include determining that the usage of the particular accelerator resource does not satisfy the idle threshold (e.g., is less than, or less than or equal to, the idle threshold, depending on the implementation). As a particular example, the usage information can include accelerator resource usage information (e.g., a utilization metric), and determining whether usage of the particular accelerator resource satisfies the idle condition can include determining whether average accelerator resource usage (e.g., determined from the utilization metric) over a particular time period satisfies an idle threshold for average accelerator resource usage over the particular time period. Certain implementations can provide a tolerance time period that defines a delay before evaluating the idle for a compute workload that has been allocated an accelerator resource to allow the workload to launch and initialize before beginning to use the accelerator resource.

[0018] Based at least on determining that usage of the particular accelerator resource satisfies the idle condition, the scheduler can deallocate the particular accelerator resource from the particular compute workload, thereby reclaiming the particular accelerator resource and reallocating the particular accelerator resource to a pending compute workload that is waiting for an accelerator resource. In some implementations, multiple compute workloads can be pending (e.g., in a pending workload queue), and the scheduler, in determining a pending workload for which to reallocate the reclaimed accelerator resource, can take into account relative priorities among the pending compute workloads. The priorities can be specified in workload information that accompanies the workloads (e.g., in container metadata such as annotations for containers). In certain implementations, the priorities can correspond to groups of compute workloads, such as workload types, project types, departments associated with the compute workloads, and any other suitable grouping criteria.

[0019] Certain implementations can run the reclamation process substantially continuously or at another suitable regular or irregular time interval or in response to particular types of events (e.g., the scheduler receiving a new compute workload). As just one example, the scheduler can run the reclamation process as a cron job that is scheduled to run at suitable time intervals (e.g., every five to seven minutes or other suitable time interval).

[0020] The particular compute workloads from which the accelerator resources are deallocated can be handled in any suitable manner. In certain implementations, the particular compute workloads from which the accelerator resources are acquired can be placed in a pending workloads queue, which is eligible to be assigned accelerator resources along with other pending compute workloads according to an applicable scheduling policy. This approach can be referred to as non-destructive preemption, as this approach moves the compute workloads from which the accelerator resources have been reclaimed to a pending state, rather than terminating these compute workloads. Of course, the present disclosure contemplates simply terminating these compute workloads where appropriate.

[0021] Certain implementations provide flexible configuration options for administrators to set idle conditions (e.g., idle threshold) / usage thresholds, priorities, time periods, and tolerance time periods. Certain implementations provide support for both physical accelerators and virtual accelerators (e.g., pGPU and vGPU).

[0022] Certain implementations can be used with containerized environments (e.g., KUBERNETES clusters), virtualized environments, high-performance computing (HPC) environments, or other suitable computing environments to efficiently allocate and manage accelerator resources for processing compute workloads within these computing environments. For example, compute workloads in a containerized environment, such as a KUBERNETES environment using a cluster, can include a built-in scheduler. As described above, certain implementations integrate with existing containerized platforms (e.g., KUBERNETES components) and can be deployed with a default scheduler (e.g., as a scheduler plugin), allowing fine-grained control over accelerator-specific compute workload management without impacting other resource types.

[0023] Turning to the drawings, Figure 1 FIG. illustrates an example computing system 100 for managing accelerator resources, according to certain implementations. The computing system 100 can be part of a computing environment, such as a containerized environment, a virtualized environment, an HPC environment, a cloud environment, an on-premises environment, or a hybrid cloud environment, some of which can overlap. In some implementations, the computing system 100 is capable of executing compute processes in parallel, such as tasks of a workload. The computing system 100 can use a client-server architecture. In the illustrated example, the computing system 100 includes a plurality of compute nodes 102, a scheduler node 104, and a network 106. While this particular implementation of the computing system 100 is illustrated and described, the present disclosure contemplates implementing the computing system 100 in any suitable manner.

[0024] In certain implementations, the compute nodes 102 can work together to perform processing operations, such as cluster operations, HPC operations, and / or other suitable types of computing operations. For example, a workload (e.g., the workload 120 described below) can be divided into smaller segments or tasks that can be parallelized across the compute nodes 102. Processes can be executed on the compute nodes 102 to perform processing operations associated with the workload. The compute nodes 102 can be implemented using any suitable combination of hardware, firmware, and software. For example, each compute node 102 can be a standalone unit (described subsequently) equipped with a processor, memory, and the like.

[0025] A workload (also referred to as a computing workload) can include a collection of one or more electronic processing tasks organized in any suitable manner. For example, a workload can include or be part of one or more software applications, one or more containers, one or more KUBERNETES Pods, one or more virtual machines, batch jobs or batch tasks, continuous integration / continuous development (CI / CD) pipelines, serverless functions or function-as-a-service (FaaS) instances, KServe endpoints, notebook computers (e.g., JUPYTER), machine learning tasks (e.g., training and / or serving tasks), inference tasks for deployed artificial intelligence (AI) models, data analytics jobs (e.g., SPARK jobs), HPC simulations, database instances or database operations, stream processing tasks, web servers, application servers, microservices, distributed ledgers or blockchain tasks, and / or any other suitable type of processing task, some of which can overlap in type.

[0026] A workload can be executed using one or more compute nodes 102 that perform processing tasks, such as tasks for a workload that are executed in a potentially parallel manner. For example, these processing tasks can be assigned to the compute nodes 102 (e.g., by the scheduler node 104) as an execution stream that involves the compute nodes 102 executing computer code, potentially in part. To this end, the compute nodes 102 can execute one or more processes of the workload that work together to execute the workload.

[0027] The compute nodes 102 can be similar to each other or dissimilar to each other. Additional details of one compute node 102 are shown. The compute node 102 includes various hardware components. For example, the compute node 102 can include a processor 108, a memory 110, an interface 112, and one or more accelerators 114a-114n (which can be referred to singularly as an accelerator 114 or in the plural as accelerators 114). The hardware components can be interconnected by a plurality of buses and / or networks. In one example, the processor 108, the memory 110, the interface 112, and the accelerators 114 can be communicatively coupled via a bus 116, such as a PCI-Express bus.

[0028] The processor 108 retrieves executable code from the memory 110 and executes the executable code. The executable code, when executed by the processor 108, can cause the processor 108 to implement any of the functionality described herein. The processor 108 can be a microprocessor, an ASIC, a microcontroller, etc. Although referred to in the singular, the processor 108 can be a plurality of processors located in one or more locations.

[0029] The memory 110 can include various types of memory, including volatile and non-volatile memory. For example, the memory 110 can include random access memory (RAM), read only memory (ROM), a hard disk drive (HDD), etc. Different types of memory can be used for different data storage needs. For example, the processor 108 can boot from the ROM, maintain non-volatile storage in the HDD, execute program code stored in the RAM, and store data being processed in the RAM. In certain implementations, part or all of the memory 110 can be or include a database, such as one or more structured query language (SQL) servers or relational databases. The memory 110 can include a non-transitory computer-readable medium that stores instructions for execution by the processor 108. One or more modules within the compute node 102 can be embodied in part or in whole as software and / or hardware for performing any of the functionality described herein. Although referred to in the singular, the memory 110 can be a plurality of memory devices located in one or more locations.

[0030] The memory 110 can include a kernel space and a user space. The kernel space can be a reserved area of memory 110 for running the operating system kernel, kernel extensions, device drivers, etc. The user space can be an area of memory 110 for running code outside of the operating system kernel, and typically includes data for running software applications. For example, a task of a workload can be an application executed by the processor 108, and data for the workload task can be stored in the user space.

[0031] The interface 112 can be used to connect to the network 106 and communicate with other nodes (e.g., other compute nodes 102, the scheduler node 104, and / or other suitable entities) over the network 106. The interface 112 facilitates the transmission and reception of data packets between the compute node 102 and other compute nodes 102 or the scheduler node 104 (e.g., via the network 106) and can comply with one or more networking standards, such as Ethernet, Wi-Fi, etc. Although referred to in the singular, the interface 112 can be multiple interfaces.

[0032] The accelerator 114 can include a specialized processing device that can perform one or more processing tasks, such as those that can be associated with certain types of workloads. Examples of the accelerator 114 can include GPU devices, ASIC devices, FPGA devices, VPU devices, and / or other types of specialized processing devices that can be incorporated into the compute node 102 or otherwise made accessible thereto to speed up the computation of workloads. The accelerator 114 can include a streaming multi-processor. The accelerator 114 provides significant computational power to allow some tasks to be performed faster than a general-purpose processor (e.g., the processor 108).

[0033] The one or more accelerators 114 can include an exporter 118. In the illustrated example, the accelerator 114a includes the exporter 118a, the accelerator 114b includes the exporter 118b, and the accelerator 114n includes the exporter 118n. The exporters 118a-118n can be generally referred to as the exporter 118 or the plurality of exporters 118. The exporter 118 is configured to collect and report accelerator usage information. The accelerator usage information can include utilization metrics related to the corresponding accelerator 114. The accelerator utilization metrics can include accelerator compute engine utilization metrics (e.g., a percentage of time that the accelerator is processing tasks) and / or memory utilization metrics (e.g., a percentage of time that accelerator memory read / write operations are performed). In certain implementations, the accelerator usage information can include temperature (e.g., a current temperature of the accelerator 114), power consumption (e.g., an amount of power that the accelerator 114 is currently consuming), clock speed (e.g., a current speed of the accelerator cores and memory clocks), memory usage (e.g., an amount of accelerator memory that is used and available), and / or any other suitable information related to the accelerator 114.

[0034] The exporter 118 can be implemented using any suitable combination of hardware, firmware, and software. In certain implementations, the exporter 118 can be implemented as a container, daemon, or in any other suitable manner. Although each accelerator 114 is shown as including a corresponding exporter 118, the present disclosure contemplates deploying the exporter 118 in any suitable manner.

[0035] The scheduler node 104 receives workloads (now referred to as workloads 120) and assigns the workloads 120 to one or more compute nodes 102. The workloads 120 can be scheduled based on various factors, including the state and capabilities of the compute nodes 102. The scheduler node 104 can monitor the state and capabilities of the compute nodes 102 (e.g., compute utilization, memory utilization, etc.) and make workload scheduling decisions based on the state and capabilities of the compute nodes 102. All or part of certain workloads 120 can be processed using one or more accelerators 114. For example, certain workloads 120 can specifically request processing using one or more accelerators 114, certain workloads 120 can allow processing using one or more accelerators 114, while still other workloads 120 can be configured to not be suitable for processing using one or more accelerators 114. Where appropriate, the scheduler node 104 can attempt to allocate one or more accelerators 114 to workloads 120 to facilitate processing of those workloads 120.

[0036] The scheduler node 104 includes various hardware components. The scheduler node 104 can or can not include components similar to those described for the compute nodes 102 and can or can not act as a compute node (e.g., compute node 102) for processing workloads 120. In the illustrated example, the scheduler node 104 includes a processor 122, a memory 124, and an interface 126. The hardware components can be interconnected by a plurality of buses and / or networks. In one example, the processor 122, the memory 124, and the interface 126 can be communicatively coupled via a bus 128, such as a PCI-Express bus.

[0037] The processor 122 retrieves executable code from the memory 124 and executes the executable code. The executable code, when executed by the processor 122, can cause the processor 122 to implement any of the functionality described herein. The processor 122 can be a microprocessor, an application-specific integrated circuit, a microcontroller, or the like. Although referred to in the singular, the processor 122 can be a plurality of processors located in one or more locations.

[0038] Memory 124 can include various types of memory, including volatile and non-volatile memory. For example, memory 124 can include RAM, ROM, HDDs, etc. Different types of memory can be used for different data storage needs. For example, processor 122 can boot from ROM, maintain non-volatile storage in HDDs, execute program code stored in RAM, and store data being processed in RAM. In certain implementations, part or all of memory 124 can be or include a database, such as one or more SQL servers or relational databases. Memory 124 can include non-transitory computer-readable media that stores instructions for execution by processor 122. One or more modules within scheduler node 104 can be embodied, in part or in whole, as software and / or hardware for performing any of the functions described herein. Although referred to in the singular, memory 124 can be multiple memory devices located in one or more locations.

[0039] Memory 124 can include a kernel space and a user space. The kernel space can be a reserved area of memory 124 used for running operating system kernel, kernel extensions, device drivers, etc. The user space can be an area of memory 124 used for running code outside of the operating system kernel, and typically includes data used for running software applications. For example, workload scheduler can be an application executed by processor 122, and data for workload scheduler can be stored in the user space.

[0040] Interface 126 can be used to connect to network 106 and communicate with other nodes over network 106. Interface 126 facilitates the transmission and reception of data packets between scheduler node 104 and compute nodes 102 (e.g., via network 106), and can adhere to one or more networking standards, such as Ethernet, Wi-Fi, etc. Although referred to in the singular, interface 126 can be multiple interfaces.

[0041] The network 106 can be any suitable type of communication network for electronic devices, and can facilitate wired and / or wireless communication. The network 106 can communicate, for example, IP packets, frame relay frames, ATM cells, voice, video, data, and other suitable information between network addresses. The network 106 can include any suitable combination of one or more local area networks (LANs), radio access networks (RANs), metropolitan area networks (MANs), wide area networks (WANs), mobile networks (e.g., using WiMax (802.16), WiFi (802.11), 3G, 4G, 5G, or any other suitable wireless technology in any suitable combination), the whole or a part of the Internet, and / or any other communication system or systems at one or more locations, any of which can be wireless and / or wired any suitable combination. The network 106 can include controllers, APs, switches, routers, firewalls, etc. for forwarding traffic.

[0042] The network 106 can facilitate coordination and synchronization of the compute nodes 102 and the scheduler node 104 in processing the workload 120 and other related tasks. In certain implementations, some or all components of the network 106 work together to provide high-bandwidth interconnection between the compute nodes 102 and the scheduler node 104. The design of at least a portion of the network 106 can prioritize low latency and high throughput between connected components. For example, some or all of the network 106 can be based on technologies such as Ethernet, InfiniBand, etc.

[0043] The computing system 100 can include a storage device 130. While illustrated as separate from the scheduler node 104, in certain implementations, the scheduler node 104 can include the storage device 130 (e.g., as part of the memory 124). The storage device 130 can include various types of memory, including volatile and non-volatile memory. For example, the storage device 130 can include RAM, ROM, HDD, etc. In certain implementations, some or all of the storage device 130 can be or include a database, such as one or more SQL servers or relational databases. While referred to in the singular, the storage device 130 can be multiple storage devices located at one or more locations.

[0044] The storage device 130 can store workload information 132 and accelerator usage information 134. The workload information 132 can include any suitable information about the workloads 120. For example, the workload information 132 can include one or more categories of the workloads 120, one or more priorities of the workloads 120, start times of the workloads 120, and / or any other suitable information about the workloads 120. The one or more categories of the workloads 120 can include frameworks associated with the workloads 120 (e.g., KUBERNETES, SPARK, LIVY, RAY, etc.), projects associated with the workloads 120 (e.g., Project A, Project B, etc.), departments associated with the workloads 120 (e.g., billing, IT, human resources, etc.), users associated with the workloads 120 (e.g., User 1, User 2, etc.), and / or any other suitable categories of workloads. The one or more priorities can define one or more priority levels for the workloads 120, which can generally involve priorities associated with at least some other workloads 120 (e.g., other workloads 120 associated with the same tenant) in order to compare relative priorities of the workloads 120. In some implementations, the priorities can be associated with one or more categories assigned to the workloads 120. As just one example, workloads 120 associated with Project A can have a higher priority than workloads associated with Project B. The start times of the workloads 120 can identify times at which the workloads 120 are allocated resources (potentially including one or more accelerators 114) and otherwise deployed to one or more compute nodes 102 for processing, and can be updated once the workloads 120 have the ability to start processing. These examples of the workload information 132 can be useful for various reasons described in more detail below.

[0045] In certain implementations, the scheduler node 104 (or another suitable component) can determine some or all of the workload information 132 and store that workload information in the storage device 130. In certain implementations, some or all of the workload information 132 can be determined from information included in the workloads 120, such as information included in annotations / tags and / or other metadata of the workloads 120. Some or all of the workload information 132 can be determined from the way in which the scheduler node 104 processes the corresponding workloads, such as which accelerators 114 are assigned to the workloads, if any, and other related information. The workload information 132 can be stored with the workloads 120 or separately.

[0046] Accelerator usage information 134 may include any suitable information about accelerator 114, workload 120, and / or any other suitable information. For example, accelerator usage information 134 may include some or all of the accelerator usage information (e.g., utilization metrics) reported by exporter 118. Accelerator usage information 134 may include information that scheduler 104 can use to determine whether accelerator resources are being used and, if not, the length of time that accelerator resources remain unused.

[0047] In some implementations, some or all of the accelerator usage information 134 is retrieved and stored as time-series data in storage device 130. For example, some or all of the storage device 130 may be implemented as a PROMETHEUS or other suitable type of database configured to collect accelerator usage information, such as utilization metrics or other suitable usage information, from exporter 118 at regular or irregular intervals.

[0048] Although described separately, workload information 132 and accelerator usage information 134 can be stored separately or together. For example, workload information 132 and accelerator usage information 134 can be mixed together, such that workload information 132 and accelerator usage information 134 associated with a specific workload 120 and one or more assigned accelerators 114 are combined in an appropriate manner, and this information itself can be stored as separate information from workload information and / or accelerator usage information 134. Furthermore, workload information 132 and accelerator usage information 134 can be analyzed to derive additional information that may be associated with a specific workload 120 and one or more assigned accelerators 114, and the derived information can also be stored as part of workload information 132 and / or accelerator usage information 134 or separately therefrom.

[0049] As described above, scheduler node 104 can determine that certain workloads 120 can be processed wholly or partially using one or more accelerators 114 (e.g., using one or more accelerator resources). For a workload 120 that can be processed using one or more accelerators 114, scheduler node 104 can attempt to allocate one or more accelerators 114 to that workload 120 and ultimately facilitate the deployment of workload 120 to one or more appropriate compute nodes 102 for processing using the allocated one or more accelerators 114. For simplicity, it will be assumed that, to the extent that scheduler node 104 allocates accelerators 114 to workload 120, scheduler node 104 allocates a single accelerator 114 to that workload 120 and allocates the entire processing power of that single accelerator 114 to that workload. However, this disclosure contemplates that scheduler node 104 assigns any suitable number of accelerators 114 to workload 120 and assigns a portion of the processing power of the accelerators 114 to workload 120.

[0050] This disclosure uses the term "accelerator resource" in various parts of the specification. In some implementations, an accelerator resource may include one or more accelerators 114, and for each of the one or more accelerators 114, the accelerator resource may include a portion or all of the processing power of the accelerator 114. An accelerator resource may include one or more physical accelerators and / or one or more virtual accelerators. In some implementations, allocating accelerator resources to workload 120 reserves those accelerator resources for that workload 120.

[0051] Once workload 120 has been allocated resources (e.g., potentially including accelerator resources) and deployed to one or more compute nodes 102 for processing, workload 120 can be considered running. In some scenarios, while workload 120 is running, it may become idle and cease using some or all of the resources allocated to it, thus rendering accelerator resources and / or other resources already allocated to that workload 120 idle. In some implementations, scheduler node 104 may perform a reclamation process, through which scheduler node 104 can identify accelerator resources that have been allocated to workload 120 but are idle, based on a certain criterion. The reclamation process may include reassigning accelerator resources that have been allocated to workload 120 but identified as idle to other pending workloads 120. Pending workloads 120 may remain in a workload queue (or other suitable data structure) awaiting the availability of compute resources (e.g., potentially including accelerator resources) for allocation to pending workloads 120.

[0052] In an example implementation of the recycling process, scheduler node 104 can monitor the usage of accelerator resources by running workloads 120. For example, scheduler node 104 can acquire accelerator usage information 134 at appropriate intervals. Accelerator usage information 134 may include utilization metrics and / or other usage information that can be used to determine whether a particular accelerator resource has been used within a time period. In some implementations, accelerator usage information 134 may be used to determine the average usage of a particular accelerator resource within a time period (e.g., average accelerator utilization).

[0053] As an example of continuing the recycling process, based on monitoring the usage of accelerator resources allocated to running workload 120, scheduler node 104 can determine whether the usage of accelerator resources meets an idle condition. The idle condition can be designed to determine whether workload 120, which has been allocated accelerator resources, is fully utilizing those allocated resources. A specific workload 120 not meeting (failing to meet) the idle condition can indicate that the specific workload 120 has fully utilized the accelerator resources allocated to it, and the accelerator resources allocated to it will not be reassigned to pending workload 120 at this time (will not be recycled). Conversely, a specific workload 120 meeting the idle condition can indicate that workload 120 has not fully utilized the accelerator resources allocated to it, and the accelerator resources allocated to it will be reassigned to pending workload 120 (will be recycled).

[0054] This disclosure envisions implementing idle conditions in any suitable manner. In some implementations, idle conditions are implemented at least in part using an idle threshold. The idle threshold may define the amount of accelerator resource usage that workload 120 is expected to achieve, or the risk of meeting the idle condition and losing the allocation of accelerator resources. Idle conditions may include a time component that complements the idle threshold. For example, the time component may be referred to as an idle time threshold and may be a time period during which accelerator usage is considered to determine whether workload 120's use of accelerator resources meets the idle threshold. The idle threshold may be expressed as the minimum percentage of usage expected to be achieved within the time period defined by the idle time threshold.

[0055] For a specific workload 120, the average utilization of accelerator resources allocated to the specific workload 120 during the applicable time period can be obtained (e.g., potentially captured or determined by a utilization metric by accessing accelerator usage information 134) and compared to an idle threshold to determine whether the use of the accelerators allocated to the specific workload 120 meets the idle threshold. The term "satisfactory" can mean less than, less than or equal to, greater than, greater than or equal to, or equal to, depending on the implementation. For ease of description and for the purposes of the examples described throughout this disclosure, it will be assumed that the value representing the idle threshold (e.g., the average accelerator usage (e.g., utilization) during the applicable time period) is greater than the idle threshold.

[0056] Furthermore, the specific time period defined by the idle time threshold can be any suitable time period. For example, the time period can correspond to the time interval between the execution of the recycling process. As another example, the time period can correspond to the time when a specific number of accelerator usage metrics (e.g., as time series data) from accelerator 114 occur. As yet another example, the time period can be any number that is determined to be important by the system administrator or another suitable user of at least a portion of the computing system 100.

[0057] In the first example, the idle threshold can be defined as zero, and the relevant idle time threshold can be five minutes. With these parameters, if the average accelerator usage of a specific workload 120 within the time period defined by the idle time threshold (e.g., determined based on accelerator usage information 134) exceeds zero, then the specific workload 120 can meet the idle condition, thus preventing the accelerator resources already allocated to the specific workload 120 from being reassigned to pending workloads 120 (reclaimed). In other words, in this example, a zero idle threshold essentially forces a situation where workloads that do not at least partially use the accelerator resources assigned to them risk losing those resources.

[0058] If a specific workload 120 fails to meet (or fails to meet) the idle threshold, it indicates that the specific workload 120 has underutilized the accelerator resources allocated to it during the period defined by the idle time threshold, and the accelerator resources allocated to the specific workload 120 will be reallocated to pending workload 120. Conversely, if a specific workload 120 meets the idle threshold, it indicates that the specific workload 120 has fully utilized the accelerator resources allocated to it during the period defined by the idle time threshold, and the accelerator resources will not be reallocated to pending workload 120 (they will not be reclaimed).

[0059] As another example, the idle threshold can be defined as 20%, and the associated idle time threshold can be five minutes. Using these parameters, if a specific workload 120 has an average accelerator usage (e.g., determined based on accelerator usage information 134) exceeding 20% ​​within a defined time period defined by the idle time threshold, then the specific workload 120 can meet the idle condition, thus preventing the reassignment of accelerator resources already allocated to the specific workload 120 to pending workloads 120. In other words, in this example, an idle threshold value of 20% essentially forces a situation where workloads with an average accelerator resource utilization not exceeding 20% ​​within the time period defined by the idle time threshold risk losing those accelerator resources.

[0060] Based at least on the determination that the use of accelerator resources by a specific workload 120 meets idle conditions (e.g., underutilization), scheduler node 104 may reclaim accelerator resources by reallocating them to one or more other workloads 120. The reallocated accelerator resources may be referred to as reclaimed accelerator resources. In some implementations, reallocating accelerator resources to one or more other workloads may include releasing accelerator resources from the specific workload 120. For example, scheduler node 104 may send a notification to the specific workload 120 to inform it that accelerator resources are being reclaimed.

[0061] Accelerator resources being processed from a specific computational workload 120 that has been reclaimed (e.g., being deallocated) can be handled in any suitable manner. In some implementations, a specific computational workload 120 can be placed in a pending workload queue, which, depending on an applicable scheduling policy, is eligible to be assigned accelerator resources along with other pending computational workloads 120. This approach can be referred to as non-destructive preemption because it moves accelerator resources that have been reclaimed from the computational workload 120 to a pending state rather than terminating those computational workloads 120. Of course, this disclosure contemplates simply terminating these computational workloads 120 or processing them in some other way where appropriate.

[0062] One or more other workloads 120 to which accelerator resources are reallocated may be pending workloads 120, such as workloads 120 waiting for available accelerator resources in a workload queue. In the presence of multiple pending workloads 120, this disclosure contemplates any suitable technique for scheduler node 104 to determine which pending workloads 120 will be allocated reclaimed accelerator resources. The selection of which(s) pending workloads 120 will be reassigned new available accelerator resources (e.g., accelerator resources reclaimed due to idle behavior of workloads 120 previously allocated these resources) can take into account various factors. These factors may include one or more of the following: the total available accelerator resources reclaimed, the accelerator resource requirements of the pending workloads 120 (individually and in possible combinations), the relative priority of the pending workloads 120, and / or any other suitable factors. The relative importance of these (and possibly other suitable) factors may vary depending on the implementation.

[0063] As described above, in some implementations, workload 120 may have one or more assigned priorities. Scheduler node 104 may be configured to evaluate the priorities of pending workloads 120 when determining which pending workloads 120 will be allocated the reclaimed accelerator resources.

[0064] For example, an implementation can be configured such that the relative priority of pending workloads 120 is the most important factor in determining which pending workload 120 will be allocated the reclaimed accelerator resources, wherein a possible tiebreaker is the position of the pending workloads 120 with the same highest priority in the pending workload queue, such that the oldest pending workload 120 with the same highest priority will be allocated the reclaimed accelerator resources.

[0065] As another example, an implementation can be configured such that the relative priority of pending workloads 120 is the most important factor in determining which pending workload 120 will be allocated the reclaimed accelerator resources, wherein a possible disconnect is the position of pending workloads 120 with the same highest priority in the pending workload queue, such that the oldest pending workload 120 with the same highest priority will be allocated the reclaimed accelerator resources.

[0066] Different workloads 120 (pending and running) may require different numbers of accelerator resources. Consequently, in some scenarios, the amount of accelerator resources reclaimed at a given time due to idle workloads 120 may be sufficient for some pending workloads 120 (or combinations of pending workloads 120), but insufficient for other pending workloads 120 (or combinations of pending workloads 120). Scheduler node 104 can be configured to consider this information when determining which pending workloads 120 will be allocated the reclaimed accelerator resources. In examples where the reclaimed accelerator resources are insufficient for higher-priority pending workloads 120 but sufficient for lower-priority pending workloads 120, different possible configurations exist; two examples are described below.

[0067] In a first possible example configuration, even after scheduler node 104 determines that the reclaimed accelerator resources are insufficient for higher-priority pending workloads 120, the higher-priority pending workloads 120 may still be preferred over the lower-priority pending workloads 120. Therefore, scheduler node 104 can retain these reclaimed accelerator resources for future combination with other reclaimed (or even released) accelerator resources, allowing higher-priority pending workloads 120 to be allocated accelerator resources before lower-priority pending workloads 120, rather than allocating these reclaimed accelerator resources to one or more lower-priority workloads 120. In a second possible example configuration of this scenario, at least based on scheduler node 104 determining that the reclaimed accelerator resources are insufficient for higher-priority pending workloads 120 but sufficient for one or more lower-priority pending workloads 120, scheduler node 104 can allocate these reclaimed accelerator resources to one or more lower-priority workloads 120, thus preventing these reclaimed accelerator resources from remaining idle.

[0068] The recycling process can be implemented as a scheduling process (e.g., a timed task) that is scheduled to run at appropriate regular or irregular time intervals. Additionally or alternatively, the recycling process can be executed in response to a specific event. As an example only, scheduler node 104 can run the recycling process in response to a new workload 120 being added to the pending workload queue (e.g., in response to determining that the accelerator resources for processing the new workload 120 are unavailable and the new workload 120 will remain in the pending workload queue). Additional details of the example implementation of the recycling process are described throughout the remainder of this disclosure.

[0069] In some implementations, the criteria used to determine whether accelerator resources meet idle criteria and / or to determine which pending workload(s) 120 will be allocated reclaimed accelerator resources can be configurable by adjusting one or more parameters. For example, configurable parameters may include one or more of idle thresholds, idle time thresholds, tolerance periods, and prioritization information.

[0070] In some implementations, to facilitate the configurability of these parameters and monitor the status of accelerator resources and workload 120, computing system 100 may include a management interface 136, which, where appropriate, can be used to control scheduler node 104 and other components of computing system 100. System administrators or other suitable human or machine users can access scheduler node 104 using management interface 136. Management interface 136 can be a central access point for scheduler node 104, accessible from a public computer network such as the Internet. Scheduler node 104 can receive commands via management interface 136. Scheduler node 104 can process commands from management interface 136, verify commands, and execute the logic specified by the commands. Furthermore, scheduler node 104 can output the results of commands via management interface 136. Examples of management interface 136 include command-line interfaces, graphical user interfaces, network interfaces, etc.

[0071] In some implementations, management interface 136 can display information about workload 120 and the accelerators 114 used to process these workloads 120. See below for reference. Figure 7 A specific example of the display for the management interface 136 is illustrated and described.

[0072] In some implementations, management interface 136 can be used to configure / customize various parameters associated with the reclamation process performed by scheduler node 104. For example, management interface 136 can be used to specify / modify one or more aspects of idle conditions to determine whether accelerator 114 is idle, change the priority of workload 120, change the category of workload 120, and / or perform other appropriate operations. As a specific example, regarding idle conditions, management interface 136 can be used to specify / modify idle thresholds, idle time thresholds, and / or other appropriate information.

[0073] Continue to refer to Figure 1Computing node 102 and scheduler node 104 may include any suitable combination of hardware, firmware, and software that can cooperate to provide the features of computing system 100. Furthermore, where appropriate, each of computing node 102 and scheduler node 104 may include one or more computer systems located at one or more locations. Each computer system may include any suitable input device, output device, mass storage medium, processor, memory, or other suitable components for receiving, processing, storing, and transmitting data. Although illustrated and described separately, computing node 102 and scheduler node 104 may be combined or further separated in any suitable manner. For example, these components may be implemented using one or more computing devices located at one or more geographical locations. Therefore, the implementations disclosed herein should not be limited to... Figure 1 The component configuration shown.

[0074] This disclosure envisions a recycling process that can be used with any suitable type of computing system. For example, a recycling process can be used with any suitable type of computing system where resources can be allocated to specific resource-using entities that can allow the allocated resources to remain idle while other resource-using entities can be waiting for resources.

[0075] Figure 2 Additional details of scheduler node 104, based on certain implementations, are illustrated. For example, Figure 2 Additional details of a computer system configured to implement scheduler node 104, according to certain implementations, are illustrated. In the example shown, as referenced above... Figure 1 As described in detail, the scheduler node 104 includes a processor 122, a memory 124, an interface 126, and a bus 128.

[0076] Returning to memory 124, in the example shown, memory 124 stores workload queue 200 and scheduler 202. Each of these will be described in more detail below.

[0077] Workload queue 200 may be a data structure for storing pending workloads 120. Although described as a queue, workload queue 200 may be any suitable data structure. Although described as storing workloads 120, workload queue 200 may store one or more of the following: the actual workload 120, a pointer to workload 120, information selected from workload 120, and / or any other suitable information about workload 120. See below for reference. Figure 5 The example workload queue 200 is described in more detail.

[0078] Scheduler 202 can represent a set of instructions and information that configure a scheduler node to perform scheduling operations, including the recycling process described herein. In the example shown, scheduler 202 includes scheduler logic 204, recycling logic 206, and recycling parameters 208. Although the scheduler 202 shown and described in the figures includes these specific items, scheduler 202 may include these and / or different items. Furthermore, while the items of scheduler 202 are shown to be separated or combined in a particular manner, other configurations are possible. References below... Figure 3 and Figure 4 Two example configurations for scheduler 202 are described.

[0079] Continue to refer to Figure 2 Scheduler logic 204 may represent instructions for scheduling workload 120, while recycling logic 206 represents instructions for implementing the recycling process and related scheduling described herein. For example, among other features, recycling logic 206 may include logic for: monitoring accelerator resources, determining which accelerator resources meet idle conditions, and reallocating accelerator resources that have been determined to meet idle conditions.

[0080] The recycling logic 206 can use recycling parameters 208 to perform the recycling process. In the example shown, recycling parameters 208 include one or more idle thresholds 210, one or more idle time thresholds 212, one or more tolerance time periods 214, and prioritization information 216. Although recycling parameters 208 are shown as including specific parameters, they may include these and / or other parameters where appropriate.

[0081] Although shown in plural form, different recycling parameters 208 can be represented in singular or plural form. In some implementations, different recycling parameters 208 may be defined for different categories of workloads 120, for different tenants of the computing environment (e.g., computing system 100), and / or for other reasons. Each of these example recycling parameters 208 is described below.

[0082] The idle threshold 210 can define the expected usage of accelerator resources by workload 120, or the risk that scheduler node 104 will reclaim accelerator resources (e.g., lose allocation of accelerator resources). While this disclosure contemplates evaluating any suitable parameter (or combination of parameters) to measure the "usage" of accelerator resources, in some implementations, the idle threshold 210 can be expressed as a percentage of usage. The idle threshold 210 can be expressed as the minimum percentage of usage expected to be achieved. For example, as a utilization metric, 0% can indicate that the accelerator resources are idle during the measured time period, while 100% can indicate that the accelerator resources are fully utilized during the measured time period. In such examples, the idle threshold 210 can be set to 0%, 20%, 40%, or any other suitable percentage. In some implementations, a higher idle threshold 210 establishes a higher accelerator utilization to avoid being characterized as idle.

[0083] Idle time threshold 212 represents the time component of the idle condition. This time component can complement idle threshold 210. For example, idle time threshold 212 can be a time period during which accelerator resource usage is considered to determine whether workload 120's use of accelerator resources meets idle threshold 210. Idle threshold 210 can be expressed as the minimum percentage of usage expected to be achieved within the time period defined by idle time threshold 212. As a specific example, if the idle threshold is 0% and idle time threshold 212 is 300s, then workload 120's use of accelerator resources should be greater than 0% within the relevant 300s time period to avoid the accelerator resources being characterized as idle.

[0084] The tolerance period 214 can be defined as a delay before assessing the idleness of the workload 120, which has been allocated accelerator resources, to allow the workload 120 to start and initialize before using the accelerator resources. The start time of the workload 120 can be determined from the workload information 132 or another suitable source, and the tolerance period 214 can be calculated based on the start time. After the tolerance period 214 expires, the idleness of the workload 120 and its allocated accelerator resources, as well as the possible reclamation of the accelerator resources, can be assessed.

[0085] Prioritization information 216 may include information identifying the relative priority levels assigned to different categories for workload 120. As described above, workload information 132 may include one or more categories of workload 120. Prioritization information 216 may include a mapping between these categories and the priorities assigned to a particular category.

[0086] Figure 3 The illustration shows an example scheduler 200a based on certain implementations. Scheduler 200a represents... Figure 2 Possible implementations of scheduler 202. InFigure 3 In the example shown, scheduler 202a includes controller 300a and scheduler logic 204, each of which can be implemented using any suitable combination of hardware, firmware, and software.

[0087] Controller 300a may be a core component of the underlying computing environment software through which computing system 100 operates. By way of a few examples only, controller 300a may be a core component of software and related services used to implement virtualized environments, clustered environments, container environments, and / or any other suitable type of computing environment. In some implementations, controller 300a may operate at the control plane level of the computing environment (e.g., computing system 100).

[0088] As referenced above Figure 2 The scheduler logic 204 can represent instructions for scheduling workload 120, while the reclamation logic 206 represents instructions for implementing the reclamation process described herein. Figure 3 In the example shown, recycling logic 206 is part of scheduler logic 204 of scheduler 202a.

[0089] Some workloads 120 may be suitable for processing using accelerator resources, while others may not. In the example shown, the indicator "(No AR)" is used to identify workloads 120 that are not suitable for processing using accelerator resources, while the indicator "(AR)" is used to identify workloads 120 that are suitable for processing using accelerator resources. Figure 3 In the implementation shown, workloads 120 can be directed to scheduler 202a regardless of whether these workloads 120 are suitable for processing using accelerator resources, because scheduler logic 204 of scheduler 202a includes recycling logic 206 for performing the recycling process of this disclosure when appropriate.

[0090] Figure 4 The illustration shows an example scheduler 202b based on certain implementations. Scheduler 202b represents... Figure 2 Possible implementations of scheduler 202. In the example shown, scheduler 202b includes scheduler 302 and scheduler plugin 304.

[0091] Scheduler 202b can be considered as providing services to the computing environment (e.g., Figure 1 The default scheduler associated with the computing system 100. Scheduler 202b can be the default scheduler of the computing system 100 (see [reference]). Figure 1 The scheduler provided by the framework on which it operates. As an example only, scheduler 202b could be a KUBERNETES scheduler that provides scheduling operations within the context of the KUBERNETES system.

[0092] like Figure 4 As shown, the scheduler 302 may include a controller 300b(1) and scheduler logic 204b(1), each of which may be implemented using any suitable combination of hardware, firmware, and software. The controller 300b(1) may be similar to the one referenced above. Figure 3 The controller 300a is mentioned above. Figure 4 The scheduler logic 204b(1) in the code can provide the default scheduling functionality of scheduler 202b. For example, Figure 4 The scheduler logic 204b(1) can provide communication with computing system 100 (see...) Figure 1 The default scheduling operation associated with the framework (e.g., KUBERNETES) on which it operates.

[0093] exist Figure 4 In the example, the reclamation and associated scheduling features of scheduler 202b are provided via scheduler plugin 304, which is inserted into and operates with scheduler 302 (the default scheduler). The reclamation and associated scheduling features of scheduler plugin 304 complement the default scheduling features of scheduler 302. Figure 4 In the example, the scheduler plugin 304 includes a controller 300b(2) and scheduler logic 204b(2), which includes recycling logic 206.

[0094] Controller 300b(2) can be similar to the reference above. Figure 3 The controller 300a and the above reference Figure 4 The scheduler 302 is described in the controller 300b (1). The scheduler logic 204b (2) can provide the ability to schedule workloads suitable for processing using accelerator resources. The reclamation logic 206 represents the instructions for implementing the reclamation process described herein.

[0095] As described above, some workloads 120 may be suitable for processing using accelerator resources (indicated by "(AR)"), while other workloads 120 may not be suitable for processing using accelerator resources (indicated by "(No AR)"). In some implementations of scheduler 202b, candidate workloads 120 that are allocated accelerator resources for processing at least a portion of workload 120 may be modified to invoke scheduler plugin 304, as scheduler plugin 304 allows for additional reclamation capabilities. The techniques used to modify workloads 120 to invoke scheduler plugin 304 can vary depending on the type of computing environment. In one example in KUBERNETES, where workload 120 may include pods, the configuration / runtime specifications of the pods may be modified to set scheduler plugin 304 as the scheduler for the pods, provided that these pods are able to use accelerator resources. For example, these pods may be modified such that spec.schedulerName is set to scheduler plugin 304. Candidate workloads 120 that are not allocated to accelerator 114 may continue to be directed to scheduler 302 for scheduling. Of course, other implementations are also possible.

[0096] Figure 5 The diagram illustrates the implementation details. Figure 2 Additional details regarding an example of workload queue 200. Workload queue 200 may store pending workloads, i.e., workloads awaiting sufficient resources (which may or may not include accelerator resources) to be available for allocation. The workloads shown in workload queue 200 may be examples of workload 120 described elsewhere. Workload queue 200 may store the workloads themselves or pointers to workloads, and possibly other suitable information about the workloads.

[0097] For the purposes of this example, it will be assumed that workload queue 200 is a first-in, first-out (FIFO) queue, queuing from the right and dequeuing from the left. In this respect, the leftmost position in the workload queue is shown as position 0, and the position numbers increase to the right.

[0098] Although typically configured as a FIFO queue, scheduler node 104 (e.g., Figure 2 The scheduler 202) can analyze the workload's prioritization level or other factors when determining the next workload to be assigned to the reclaimed accelerator resources. Therefore, Figure 5 The diagram illustrates the workload ( Figure 5 The abbreviation is "WL#" in Chinese), and an indication of the priority level for that workload. Figure 5(abbreviated as "PL#" in Chinese). In this example, it is assumed that three priority levels (1, 2, and 3) are possible, where PL 1 is the highest priority and PL 3 is the lowest priority. In some implementations, scheduler node 104 may select the workload with the highest priority, resulting in the selection of the oldest workload with that priority level (e.g., based on its position in workload queue 200). Furthermore, workload numbers (e.g., 34, 26, 38, etc.) do not necessarily indicate the order in which they arrive at scheduler node 104 / workload queue 200. Instead, these numbers simply represent workload identifiers. Of course, other implementations are also possible.

[0099] In the example shown, assuming that scheduler node 104 has reclaimed accelerator resources from running workloads, and that the reclaimed accelerator resources are sufficient for any workload in workload queue 200, scheduler node 104 can analyze the relative priority levels of the workloads in workload queue 200 and determine that workloads 26 (position 1), 38 (position 2), and 39 (position 5) have the highest priority among the pending workloads in workload queue 200. In some implementations, since workload 26 is at position 1, which means it is higher than 38 or 39 in workload queue 200 and has therefore been waiting for resource allocation for a longer period, scheduler node 104 can determine that the reclaimed accelerator resources will be reallocated to workload 26.

[0100] Figure 5 The description assumes a scenario where the applicable idle threshold is consistent across all compute workloads in workload queue 200. More complex implementations are possible. For example, specific idle thresholds can be defined for certain categories of compute workloads, while different idle thresholds (one or more) can be defined for other categories. In such an implementation, determining which compute workload to allocate reclaimed accelerator resources to can include more complex determinations.

[0101] Figure 6 The illustration shows an example of a workload information table 600 according to some implementation, which may include... Figure 1 The workload information 132 and / or a portion thereof. In the example shown, the workload information table 600 includes multiple columns 602a-602j (collectively referred to as columns 602) and multiple rows 604a-604f (collectively referred to as rows 604). Column 602 corresponds to a specific type of information, and row 604 corresponds to a specific workload (e.g., Figure 1 Workload 120). Although Figure 6Information stored in a tabular format is shown (e.g., in workload information table 600), but this disclosure contemplates storing workload information 132 in any suitable format. The contents of the different columns 602 will now be described.

[0102] Column 602a indicates the workload identifier (ID), where the workload is abbreviated as WL in the header. In this example, the workload ID is an integer, but any suitable ID can be used.

[0103] Columns 602b through 602d identify the different category types to which workloads can be assigned. For example, columns 602b, 602c, and 602d correspond to category 1 (CAT.1), category 2 (CAT.2), and category 3 (CAT.3), respectively. This disclosure contemplates assigning workload 120 to any suitable number and type of categories, excluding any categories if appropriate for a particular implementation. In this example, category 1 specifies the framework associated with the workload, category 2 specifies the department associated with the workload, and category 3 specifies the project associated with the workload.

[0104] Column 602e identifies the status of the workload. This status can indicate whether workload 120 is running. As mentioned earlier, a running workload 120 can be one that has been assigned resources (and possibly accelerator resources) and deployed (e.g., deployed to...). Figure 1 One or more compute nodes 102 are used for the execution of workloads. In this example, workloads 5 (line 604a) and 28 (line 604d) are running. The remaining workloads are shown as “Not Applicable” or “N / A” because, as described below, these workloads are waiting for resource (potentially accelerator resources) allocation in a workload queue (e.g., workload queue 200).

[0105] For those workloads 120 that are waiting for pending workloads in the workload queue (e.g., workload queue 200), column 602f indicates the workload queue position of the workload. In this example, workloads 12 (row 604b), 19 (row 604c), 31 (row 604e), and 43 (row 604f) show workload queue positions 3, 1, 7, and 2, respectively. Workloads 5 (row 604a) and 28 (row 604d) show an "N / A" status because, as previously stated, workloads 5 and 28 are running and are not waiting in the pending workload queue.

[0106] For those workloads 120 that have already been allocated accelerator resources, column 602g indicates one or more identifiers of the accelerator resources that have been allocated to workload 120. In this example, workloads 5 (row 604a) and 28 (row 604d) are running (see column 602e) and have been allocated accelerator resources identified by their respective AR IDs. The remaining workloads are shown as "N / A" because, as previously stated, these workloads are waiting for resource (potentially accelerator resources) allocation in a workload queue (e.g., workload queue 200).

[0107] Column 602h indicates the start time of those 120 workloads that are currently running. In this example, workloads 5 (row 604a) and 28 (row 604d) are running (see column 602e), and their start times are indicated as time 1 and time 2, respectively. The remaining workloads are shown with start times of "N / A" because, as previously mentioned, these workloads are waiting for resource (potentially accelerator resources) allocation in the workload queue (e.g., workload queue 200). The start times indicated in column 602h can be used to assess tolerable time periods (e.g., Figure 2 Whether the tolerance period 214 has expired, in order to assess whether workloads 5 and 28 may meet the idle conditions.

[0108] Column 602i indicates the priority level (abbreviated as PL) assigned to workload 120. In the example shown, workload information table 600 includes three priority levels (1, 2, and 3) for workload 120, and each workload 120 is shown as including only one priority level. As described elsewhere, priority levels can be associated with the category to which workload 120 is associated, and workload 120 can be associated with multiple priority levels.

[0109] Column 602j indicates the idle threshold applicable to each workload of 120 ( Figure 2 The idle threshold 210). In the example shown, each workload 120 has an idle threshold of zero.

[0110] Although not shown, additional information in the workload information table 600 may include idle time thresholds (e.g., idle time threshold 212) and / or tolerance time periods 214 applicable to workload 120. Because these parameters may vary from workload to workload in some implementations, it may be useful to store these values ​​in association with workload 120 (e.g., in workload information table 600) so that applicable parameter values ​​can be determined and used.

[0111] Figure 7 The illustration shows a sample user interface 700 according to some implementations. In some implementations, user interface 700 may be...Figure 1 The management interface 136 generates an example of at least one interface for managing the scheduler node 104. The management interface 136 and / or the scheduler node 104 may use workload information 132 and / or accelerator usage information 134 to generate the user interface 700. The specific design, layout, and content of the user interface 700 are provided as examples only.

[0112] In the example shown, the user interface 700 is arranged by category. For example, a user can specify that the user interface 700 is displayed according to a specific category by selecting a category from the drop-down menu 702. In this example, the category "Frames" has been selected, and the information in the user interface 700 is arranged according to the frame category.

[0113] In the example shown, for different frameworks, the user interface 700 includes information indicating the number of assigned accelerators, their status, priority level, idle threshold, idle time threshold, and action column. In some implementations, the user interface 700 provides modification of one or more parameters via a gear icon shown in the action column (e.g., Figure 2 The ability to recycle parameters 208). Furthermore, because the user interface is arranged according to categories (e.g., frame categories), users can modify one or more parameters (e.g., ...) across all or some categories via user interface 700. Figure 2 The recycling parameter 208). For example, users can be able to change the priority level (priority level in column 602i), idle threshold (…) across the entire workload category. Figure 2 The idle threshold 210 and / or Figure 6 The idle threshold in column 602j) and / or idle time threshold ( Figure 2 The idle time threshold (212).

[0114] Figures 8 to 10 The illustrations depict various example methods according to certain implementations of this disclosure. In some implementations, with Figures 8 to 10 Some or all of the operations associated with the method are executed by scheduler node 104. For example, with Figures 8 to 10 Some or all of the operations associated with the method can be executed by scheduler node 104. For example, with Figures 8 to 10 Some or all of the operations associated with the method can be performed by scheduler 202 (including schedulers 202a and / or 202b), scheduler logic 204, and / or recycling logic 206. Furthermore, examples are described using the foregoing figures. Figures 8 to 10 This disclosure is not limited to such an implementation.

[0115] For reference Figures 8 to 10The method described herein assumes that any reclaimed accelerator resources are sufficient to handle any pending workload 120. In some implementations, it is appropriate that the scheduler node 104 considers the sufficiency of the reclaimed accelerator resources when determining which pending workload 120 to allocate them to, as referenced above. Figure 1 A more detailed description follows. Furthermore, as described above, accelerator resources may include one or more accelerators 114, and for each of the one or more accelerators 114, the accelerator resources may include a portion or all of the processing power of the accelerator 114. Accelerator resources may include one or more physical accelerators and / or one or more virtual accelerators. In some implementations, allocating accelerator resources to workload 120 reserves those accelerator resources for that workload 120.

[0116] Figure 8 The illustration shows an example method 800 for managing accelerator resources, based on certain implementations. Method 800 may be referred to as a reclamation process and can be configured to automatically and dynamically reclaim accelerator resources from workloads and reassign these reclaimed accelerator resources to pending workloads. In some implementations, method 800 may be implemented as part of a scheduler or scheduler plugin and may run as a scheduled task. Example steps of method 800 are described below.

[0117] In step 802, scheduler node 104 may monitor the usage of a first accelerator resource allocated to a first workload 120, which may also be referred to as a computing workload. The first workload 120 may be a running workload 120, including the allocation of the first accelerator resource. In some implementations, the first workload 120 may be one of a plurality of workloads 120, and scheduler node 104 may monitor the usage of the accelerator resource allocated to the respective workload 120 among the plurality of running workloads 120. In some implementations, monitoring the usage of the accelerator resource allocated to workload 120 may include scheduler node 104 receiving accelerator usage information 134 from the accelerator resource (e.g., from accelerator 114).

[0118] In step 804, scheduler node 104 may determine, based on monitoring the usage of the first accelerator resource allocated to the first computing workload 120, that the usage of the first accelerator resource allocated to the first computing workload 120 meets the idle condition. This disclosure contemplates determining whether the usage of the accelerator resource meets the idle condition in any suitable manner, and various options are described throughout this disclosure.

[0119] In some implementations, determining whether the use of accelerator resources assigned to the running workload 120 meets the idle condition may include scheduler node 104 determining whether the use of the accelerator resources meets an idle threshold. For example, scheduler node 104 may determine, based on monitoring the use of accelerator resources allocated to workload 120, that the use of a first accelerator resource allocated to the first workload 120 meets the idle condition by: determining that the use of the first accelerator resource does not meet an idle threshold (e.g., ...). Figure 2 The idle threshold (210) can have any suitable value and, in some implementations, can be expressed as a percentage. In a particular example, the value of the idle threshold is 0. Furthermore, as previously mentioned, an idle time threshold (e.g., idle time threshold 212) can be included as part of the idle condition. In some implementations, the idle condition takes into account the average accelerator utilization over a time period.

[0120] In some implementations, determining that the usage of the first accelerator resource does not meet the idle threshold may include the scheduler node 104 accessing accelerator usage information 134 of the first accelerator resource and determining, based on the accelerator usage information 134 of the first accelerator resource, whether the average accelerator usage within a time period (e.g., idle time threshold 212) meets the idle threshold (e.g., idle threshold 210). At least based on determining that the average accelerator usage of the first accelerator resource within the time period does not meet the idle threshold (e.g., not greater than the idle threshold), the scheduler node 104 may determine that the usage of the first accelerator resource allocated to the first workload 120 meets the idle condition and should be considered idle.

[0121] As described above with reference to step 802, scheduler node 104 can monitor the usage of accelerator resources (with respect to the allocation of one or more accelerator resources) by multiple running workloads 120. Scheduler node 104 can evaluate some or all other running workloads to determine whether these computing workloads meet the idle condition. For example, before and / or after determining in step 804 that the usage of a first accelerator resource allocated to a first computing workload 120 meets the idle condition, scheduler node 104 can determine whether the usage of one or more other workloads 120 on one or more other accelerator resources meets the idle condition.

[0122] In step 806, scheduler node 104 may, at least based on determining that the use of the first accelerator resource allocated to the first computing workload 120 meets an idle condition, reallocate the first accelerator resource to the second workload 120. The second workload 120 may be a pending workload 120 and may be stored in a pending workload queue (e.g., workload queue 200).

[0123] In some implementations, the second workload 120 is one of a plurality of pending workloads 120. Scheduler node 104 can use any suitable technique to determine which pending workload 120 will be allocated the first accelerator resources reclaimed from the first compute workload 120. For example, scheduler node 120 may simply select the next pending workload as the second workload 120 from the pending workload queue (e.g., workload queue 200) to allocate the first accelerator resources reclaimed from the first compute workload 120 to it. As another example, when determining which pending workload 120 will be allocated the first accelerator resources reclaimed from the first compute workload 120, scheduler node 104 may consider the relative priorities of the pending workloads 120.

[0124] Regarding the prioritization approach, in some implementations, scheduler node 104 can access a pending workload queue (e.g., workload queue 200) that includes multiple pending workloads 120, and obtain prioritization information for the pending workloads 120 in the pending workload queue. The priority information can be obtained from workload information 132, the workload 120 itself, and / or any other suitable source. Scheduler node 104 can determine a selected pending workload 120 as a second computational workload 120 based on the corresponding priority of the pending workload 120. As described in more detail elsewhere in this specification, in some implementations, the priority identified by the prioritization information corresponds to a category of workload 120.

[0125] In some implementations, reallocating reclaimed accelerator resources to the second workload 120 may include deallocating accelerator resources from the first workload 120. Accelerator resources already reclaimed from the first workload 120 can be handled in any suitable manner. In some implementations, the scheduler node 104 may, in response to reallocating accelerator resources to the second workload 120, move the first workload 120 to a pending state. This may include placing the first workload 120 in a pending workload queue, which, according to an applicable scheduling policy, is eligible to be assigned accelerator resources along with other pending compute workloads 120. This approach may be referred to as non-destructive preemption because it moves accelerator resources already reclaimed from the compute workload 120 to a pending state rather than terminating those compute workloads 120. Of course, this disclosure contemplates simply terminating these compute workloads 120 or handling them in some other way where appropriate.

[0126] Figure 9The illustration depicts an example method 900 for managing accelerator resources according to some implementation. In step 902, a reclamation process may be initiated to determine whether accelerator resources should be reclaimed and reallocated from running workloads. In some implementations, as described above, this reclamation process may be a scheduled task or other suitable type of program that the scheduler node 104 runs at regular or irregular intervals or in response to specific events.

[0127] In step 904, scheduler node 104 can determine whether any workload 120 is pending. Pending workload 120 can be a workload waiting for resources (which may include one or more accelerator resources) to become available for allocation to the pending workload 120. Pending workload 120 can be stored in a workload queue (e.g., ...). Figure 2 and Figure 5 The workload queue 200 is used to determine if any pending workloads 120 exist. In some implementations, the scheduler node 104 can access the workload queue 200 to determine if any pending workloads 120 exist.

[0128] If scheduler node 104 determines in step 904 that there are no pending workloads 120, the recycling process can terminate and return to step 902 to restart at the appropriate time and / or in response to a suitable event. Alternatively, if scheduler node 104 determines in step 904 that one or more workloads 120 are pending, method 900 can proceed to step 906.

[0129] In step 906, scheduler node 104 can determine whether all accelerator resources have been allocated. In other words, scheduler node 104 can determine whether any accelerator resources are available for the pending workload 120 identified in step 904. If scheduler node 104 determines in step 906 that not all accelerator resources have been allocated (accelerator resources are available for allocation), then method 900 can proceed to step 908. In step 908, scheduler node 104 can allocate the available accelerator resources to the pending workload 120. On the other hand, if scheduler node 104 determines in step 906 that all accelerator resources have been allocated (accelerator resources are not available for allocation to the pending workload 120), then method 900 can proceed to step 910.

[0130] In step 910, scheduler node 104 determines whether any accelerator resources already allocated to the running workload 120 are idle. In other words, in step 910, scheduler node 104 may attempt to identify idle allocated accelerator resources for reclamation. As described above, scheduler node 104 may determine whether the use of accelerator resources already allocated to the running workload satisfies an idle condition. In some implementations, scheduler node 104 uses one or more of accelerator usage information 134, an idle threshold 210, and an idle time threshold 212 to determine whether any accelerator resources already allocated to the running workload 120 are idle. Various techniques for determining whether accelerator resources are idle are described throughout this disclosure.

[0131] If scheduler node 104 determines in step 910 that no idle accelerator resources exist, the reclamation process can terminate and return to step 902 to restart at an appropriate time and / or in response to a suitable event. Alternatively, if scheduler node 104 determines in step 910 that idle allocated accelerator resources exist, method 900 can continue to step 912.

[0132] In step 912, scheduler node 104 can select one or more pending workloads 120, which will be allocated to the idle resources identified in step 910. Various techniques for selecting pending workloads 120 to receive idle accelerator resources are described throughout this disclosure. Factors may include position in the pending workload queue 200 (which may also reflect the relative length of time the workload 120 has been pending), priority of the pending workload 120, and any other suitable factors.

[0133] In step 914, scheduler node 104 may reassign one or more accelerator resources identified as idle (e.g., in step 910) to one or more pending workloads 120 selected in step 912.

[0134] In step 916, scheduler 104 can process workloads 120 from which idle accelerator resources have been reclaimed. Workloads 120 from which idle accelerator resources have been reclaimed (e.g., deallocated) can be processed in any suitable manner. In some implementations, workloads 120 can be placed in a pending workload queue (e.g., Figure 2 and Figure 5In the workload queue 200, the queue is eligible to be assigned accelerator resources along with other pending workloads 120, according to an applicable scheduling policy. This approach can be referred to as non-destructive preemption because it moves accelerator resources that have been reclaimed from the workloads 120 to a pending state, rather than terminating those workloads 120. This disclosure also envisions simply terminating accelerator resources from those workloads 120 that have been reclaimed, or otherwise handling those workloads 120, where appropriate.

[0135] Figure 10 The illustration shows an example method 1000, based on some implementation, for determining whether to reclaim accelerator resources already allocated to a specific workload 120. As an example only, method 1000 can provide methods for performing... Figure 8 Method 800, step 804 and / or Figure 9 Method 900 employs some or all of the specific techniques in step 910. Specifically, method 1000 provides a technique for traversing running workloads 120 to identify running workloads 120 that are idle relative to the accelerator resources already allocated to those running workloads 120, thereby making those allocated accelerator resources idle and available for reassignment to other pending workloads 120. For the purposes of this example, it will be assumed that the idle conditions to be evaluated by scheduler node 104 include an idle threshold (e.g., Figure 2 The idle threshold 210 and / or Figure 6 (602j).

[0136] In step 1002, scheduler node 104 may select to run workload 120 for analysis. In some implementations, scheduler node 104 may use workload information 132 (see [link to implementation]). Figure 1 This determines the running workload that has been assigned accelerator resources, for example, as follows: Figure 6 The workload information is shown in Table 600.

[0137] In step 1004, scheduler node 104 can determine whether the tolerance period for the running workload 120 selected in step 1002 has expired.

[0138] If scheduler node 104 determines in step 1004 that the tolerance period for the running workload 120 selected in step 1002 has not yet expired, method 1000 may return to step 1002 so that scheduler node 104 can select the next running workload for evaluation. However, if scheduler node 104 determines in step 1004 that the tolerance period for the running workload 120 selected in step 1002 has expired, method 1000 may continue to step 1006.

[0139] In step 1006, scheduler node 104 can access accelerator resource usage information (e.g., accelerator resource usage information) allocated to one or more accelerator resources selected in step 1002 for the running workload 120. Figure 1 (Accelerator usage information 134 in the text). In some implementations, accelerator usage information can indicate whether the running workload 120 selected in step 1002 has been consistently using one or more accelerator resources. As a specific example, accelerator usage information can specify, or can provide sufficient information to the scheduler node 104 to determine, the average accelerator usage information of one or more accelerator resources, indicating the average usage of these accelerator resources by the selected running workload 120 over a specific time period. The specific time period can be an idle time threshold (e.g., ...). Figure 2 The idle time threshold (212).

[0140] In step 1008, scheduler node 104 can determine whether the selected running workload 120's use of one or more accelerator resources meets an idle threshold based on the accelerator resource usage information accessed in step 1006. For example, scheduler node 104 can determine whether the average accelerator usage within a time period (e.g., an idle time threshold) meets the idle threshold.

[0141] As described above, the idle threshold can be adjusted to set different sensitivities to idle accelerator resources. For example, an idle threshold defined as zero essentially forces a situation where a specific workload 120 can meet the idle condition by using one or more accelerator resources at least partially during a specific time period. As another example, an idle threshold defined as 20% can force a situation where a specific workload 120 can meet the idle condition by using one or more accelerator resources for at least 20% of the time during a specific time period.

[0142] If scheduler node 104 determines in step 1008 that the selected running workload 120's use of one or more accelerator resources meets the idle threshold, then method 1000 can return to step 1002 so that scheduler node 104 selects the next running workload 120 for evaluation. Alternatively, if scheduler node 104 determines in step 1008 that the selected running workload 120's use of one or more accelerator resources does not meet the idle threshold, then method 1000 can continue to step 1010. In step 1010, based on the determination in step 1008, scheduler node 104 can determine that one or more accelerator resources allocated to the selected running workload 120 will be reclaimed from the selected running workload 120 and reassigned to pending workloads 120.

[0143] Figure 11 A block diagram of an example computing device 1100 according to some implementation is shown. As described above, implementations of this disclosure can be implemented using a computing device. For example, Figures 1 to 10 All or any part of the components or methods shown (e.g., computing system 100, computing node 102, scheduler node 104, scheduler 202 (including scheduler 202a and / or scheduler 202b), and methods 800 to 1000) may be implemented at least in part using one or more computing devices (such as computing device 1100).

[0144] The computing device 1100 may include one or more computer processors 1102, non-persistent storage devices 1104 (e.g., volatile memory such as RAM, cache memory, etc.), persistent storage devices 1106 (e.g., hard disks, optical disc drives such as optical disc (CD) drives or digital versatile disc (DVD) drives, flash memory, etc.), communication interfaces 1112 (e.g., Bluetooth interfaces, infrared interfaces, network interfaces, optical interfaces, etc.), input devices 1110, output devices 1108, and many other elements and functions. Each of these components will be described in detail below.

[0145] In some implementations, the (multiple) computer processors 1102 may be integrated circuits for processing instructions. For example, the (multiple) computer processors may be one or more cores or microkernels of a processor. Processor 1102 may be a general-purpose processor configured to execute program code included in software executed on computing device 1100. Processor 1102 may also be a special-purpose processor, where certain instructions are incorporated into the processor design. Although... Figure 11 Only one processor 1102 is shown, but computing device 1100 may include any number of processors.

[0146] The computing device 1100 may also include one or more input devices 1110, such as a touchscreen, keyboard, mouse, microphone, touchpad, electronic pen, motion sensor, or any other type of input device. The input devices 1110 allow users to interact with the computing device 1100. In some implementations, the computing device 1100 may include one or more output devices 1108, such as a screen (e.g., a liquid crystal display (LCD), plasma display, touchscreen, cathode ray tube (CRT) display, projector, or other display device), printer, external storage device, or any other output device. The one or more output devices may be the same as or different from the input devices(s). The input and output devices(s) may be locally or remotely connected to the computer processor(s) 1102, non-persistent storage device(s) 1104, and persistent storage device(s) 1106. Many different types of computing devices exist, and the aforementioned input and output devices(s) may take other forms. In some cases, a multi-mode system allows users to provide multiple types of input / output to communicate with the computing device 1100.

[0147] Furthermore, the communication interface 1112 can facilitate the connection of the computing device 1100 to a network (e.g., LAN, WAN), such as the Internet, mobile networks, or any other type of network, and / or connection to another device, such as another computing device. The communication interface 1112 can use wired and / or wireless transceivers to perform or facilitate the reception and / or transmission of wired or wireless communications, including transceivers using audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, etc. Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs Wireless signal transmission Low-power (BLE) wireless signal transmission Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), global microwave access interoperability (WiMAX), IR communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof.

[0148] The communication interface 1112 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing device 1100 based on the reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware arrangement, and therefore, the basic features described herein can be easily replaced when developing improved hardware or firmware arrangements.

[0149] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media on which data may be stored, and such media does not include carrier waves and / or transient electronic signals propagated wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or memory devices. Code and / or machine-executable instructions may be stored on a computer-readable medium, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0150] All or any part of the components of computing device 1100 may be implemented in a circuit system. For example, these components may include electronic circuits or other electronic hardware and / or be implemented using them, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), CPU, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof and / or be implemented using them to perform the various described operations. In some aspects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc.

[0151] It should be understood that the systems and methods described in this disclosure can be combined in any suitable manner.

[0152] Some implementations of this disclosure may provide some, no, or all of the following technical advantages. Some implementations can improve accelerator utilization in a computer system. For example, some implementations can improve accelerator utilization by identifying idle accelerator resources and reallocating these idle accelerator resources to pending computational workloads, which can provide dynamic and intelligent resource allocation. For example, some implementations dynamically adjust accelerator resource allocation based on usage patterns. Detecting when accelerator resources are underutilized and reallocating those resources to pending computational workloads waiting for accelerator resources can improve overall system efficiency. Furthermore, this approach can improve user experience. For example, by efficiently managing accelerator resources, some implementations can reduce the waiting time for users seeking access to accelerator resources, which can improve productivity and user satisfaction.

[0153] Some implementations can provide priority-based scheduling, such as allowing accelerator resources to be reclaimed based on the relative priority of compute workloads, potentially allowing fine-grained control over resource allocation based on configurable priorities. This approach can facilitate high-priority compute workloads or critical projects gaining access to accelerator resources when appropriate, even if doing so means preempting lower-priority idle compute workloads.

[0154] Some implementations allow users (e.g., IT administrators) to configure and tune (e.g., via a user interface) the calculation of workload priorities and / or one or more idle thresholds (e.g., time thresholds, usage thresholds, etc.) to determine whether the accelerator is idle. This configuration capability can allow accelerator resources to be shared in a way that achieves specific organizational goals, which may vary over time or across different time periods.

[0155] Some implementations allow for non-destructive preemption of compute workloads. For example, if a compute workload is preempted—meaning accelerator resources are reallocated to another compute workload—the preempted workload can be moved to a pending state instead of being terminated. This can allow the preempted compute workload to be resumed when accelerator (or other) resources become available again, potentially preserving work and improving the user experience.

[0156] Some implementations offer seamless integration with KUBERNETES and / or other containerization platforms. For example, some implementations are designed to work with KUBERNETES (or other containerization) components to potentially make the solution easy to adopt without significant changes to the overall infrastructure. Some implementations can be deployed as plug-ins into the default scheduler of a containerization platform to focus on accelerator compute workloads. Some implementations can support both physical and virtual accelerators, such as both physical and virtual GPUs, which provides versatility for different types of deployment and virtualization strategies.

[0157] Certain implementations can improve the return on investment for accelerator hardware, such as by reducing or eliminating situations where expensive accelerator resources are idle while other computing workloads wait in the queue. In some implementations, costs can be saved by reducing or eliminating additional purchases of accelerator hardware to meet demand and by utilizing existing accelerator resources more efficiently.

[0158] Although this disclosure describes or illustrates specific operations in a particular order, it contemplates that these operations may occur in any suitable order. Furthermore, it contemplates that any suitable operation may be repeated once or multiple times in any suitable order. While this disclosure describes or illustrates specific operations in sequence, it contemplates that any suitable operation may occur substantially simultaneously where appropriate. Where appropriate, any suitable operation or sequence of operations described or illustrated herein may be interrupted, suspended, or otherwise controlled by another process, such as an operating system or kernel. These actions may operate within an operating system environment or as independent routines that occupy all or most of the system processing.

[0159] Although this disclosure has been described with reference to illustrative implementations, this specification is not intended to be restrictive. By referring to this specification, those skilled in the art will clearly understand various modifications and combinations of the illustrative implementations, as well as other implementations of this disclosure. Therefore, the appended claims are intended to cover any such modifications or implementations.

Claims

1. A computing device comprising: one or more processors; and one or more non-transitory computer-readable storage media storing programming for execution by the one or more processors, the programming including instructions to: monitor usage of a first accelerator resource that is allocated to a first computing workload; determine, based on monitoring the usage of the first accelerator resource that is allocated to the first computing workload, that the usage of the first accelerator resource that is allocated to the first computing workload satisfies an idle condition; and reallocate the first accelerator resource to a second computing workload based at least on determining that the usage of the first accelerator resource that is allocated to the first computing workload satisfies the idle condition, the second computing workload being a pending computing workload.

2. The computing device of claim 1, wherein: the first workload includes first workload information, the first workload information including: a category associated with the first workload; a priority associated with the first workload; and a start time associated with the first workload; and the second workload includes second workload information, the second workload information including: a category associated with the second workload; and a priority associated with the second workload.

3. The computing device of claim 1, wherein the instructions to determine that the usage of the first accelerator resource that is allocated to the first computing workload satisfies an idle condition include instructions to determine that the usage of the first accelerator resource does not satisfy an idle threshold.

4. The computing device of claim 3, wherein the instructions to determine that the usage of the first accelerator resource does not satisfy the idle threshold include instructions to: access accelerator usage information for the first accelerator resource; determine, from the accelerator usage information, whether an average accelerator usage over a time period satisfies an idle threshold; and determine, based at least on determining that the average accelerator usage over a time period does not satisfy the idle threshold, that the usage of the first accelerator resource that is allocated to the first computing workload satisfies an idle condition.

5. The computing device of claim 4, wherein the idle threshold is zero.

6. The computing device of claim 1, wherein the instructions to monitor usage of the first accelerator resource that is allocated to the first running computing workload include instructions to receive accelerator usage information from the first accelerator resource.

7. The computing device of claim 1, wherein the programming further includes instructions to determine that the first workload has run for a tolerance time period prior to reallocating the accelerator resource to the second computing workload.

8. The computing device of claim 1, wherein: the second computing workload is one of a plurality of pending computing workloads; and the programming further includes instructions to: determine, based on monitoring the usage of the first accelerator resource that is allocated to the first computing workload, that the usage of the first accelerator resource that is allocated to the first computing workload satisfies an idle condition; and reallocate the first accelerator resource to a second computing workload based at least on determining that the usage of the first accelerator resource that is allocated to the first computing workload satisfies the idle condition, the second computing workload being a pending computing workload. accessing a pending workload queue that includes the plurality of pending compute workloads; obtaining priority information for the plurality of pending compute workloads; and determining, as the second compute workload, a selected pending workload according to respective priorities of the plurality of pending compute workloads.

9. The computing device of claim 8, wherein the priority identified by the priority information corresponds to a class for the compute workload.

10. The computing device of claim 1, wherein: reassigning the accelerator resource to a second compute workload includes de-assigning the accelerator resource from the first compute workload; and the programmatic design further includes instructions to transition the first compute workload to a pending state in response to reassigning the accelerator resource to a second compute workload.

11. The computing device of claim 1, wherein the first accelerator resource is a graphics processing unit (GPU) or a portion of a GPU.

12. The computing device of claim 1, wherein: the second compute workload includes one or more containers; and the accelerator resource operates in a containerized environment.

13. A computer-implemented method comprising: monitoring, by a computing device, usage of a first accelerator resource assigned to a first compute workload; determining, by the computing device, based on monitoring the usage of the first accelerator resource assigned to the first compute workload, that the usage of the first accelerator resource assigned to the first compute workload satisfies an idle condition; and reassigning, by the computing device, based at least on determining that the usage of the first accelerator resource assigned to the first compute workload satisfies the idle condition, the first accelerator resource to a second compute workload, the second compute workload being a pending compute workload.

14. The computer-implemented method of claim 13, comprising: monitoring usage of a plurality of accelerator resources assigned to respective compute workloads of a plurality of running compute workloads, the first compute workload being one of the plurality of running compute workloads, respective the accelerator resources for the first compute workload including the first accelerator resource; prior to determining that the usage of the first accelerator resource assigned to the first compute workload satisfies the idle condition, determining that the usage of a second accelerator resource of the plurality of accelerator resources by a third workload of the plurality of running workloads does not satisfy the idle condition.

15. The computer-implemented method of claim 13, wherein determining that the usage of the first accelerator resource assigned to the first compute workload satisfies an idle condition includes determining that the usage of the first accelerator resource does not satisfy an idle threshold.

16. The computer-implemented method of claim 15, wherein determining that the usage of the first accelerator resource does not satisfy the idle threshold includes: accessing accelerator usage information for the first accelerator resource; determining, from the accelerator usage information, whether average accelerator usage over a time period satisfies an idle threshold; and based at least on determining that the average accelerator usage over a time period does not satisfy the idle threshold, determining that the usage of the first accelerator resource allocated to the first compute workload satisfies an idle condition.

17. The computer-implemented method of claim 13, wherein monitoring usage of the first accelerator resource allocated to the first running compute workload comprises receiving accelerator utilization information from the first accelerator resource.

18. The computer-implemented method of claim 13, wherein: the second compute workload is one of a plurality of pending compute workloads; and the method further comprises: accessing a pending workload queue comprising the plurality of pending compute workloads; obtaining priority information for the plurality of pending compute workloads; and determining, as the second compute workload, a selected pending workload according to respective priorities of the plurality of pending compute workloads.

19. The computer-implemented method of claim 13, wherein: re-allocating the accelerator resource to a second compute workload comprises de-allocating the accelerator resource from the first compute workload; and the method further comprises transitioning the first compute workload to a pending state in response to re-allocating the accelerator resource to a second compute workload.

20. One or more non-transitory computer-readable storage media storing programming for execution by the one or more processors, the programming including instructions to: monitor usage of a first accelerator resource allocated to a first compute workload; based on monitoring the usage of the first accelerator resource allocated to the first compute workload, determine that the usage of the first accelerator resource allocated to the first compute workload satisfies an idle condition; and based at least on determining that the usage of the first accelerator resource allocated to the first compute workload satisfies the idle condition, re-allocate the first accelerator resource to a second compute workload, the second compute workload being a pending compute workload.

Images