Selective execution of workloads using hardware accelerators

Abstract

Systems and methods for selectively executing workloads using hardware accelerators are described. The method includes the client application submitting, directly to the hardware accelerator, a command to execute a workload, where the command includes an indication of performance expectations from the hardware accelerator, and where the workload is executable by a computing core accessible to the client application or by the hardware accelerator. The method further includes, upon receiving a retry response from the hardware accelerator, the client application executing the workload using a computing core accessible to the client application, where the hardware accelerator is configured to provide the retry response directly to the client application upon determining that the hardware accelerator cannot meet the performance expectation.

Description

BACKGROUND

[0001] Client applications can run in virtual environments hosted by servers or other computers. As an example, resources including compute cores, memory, and input / output (IO) resources are allocated to virtual machines, allowing them to execute client applications using one or more compute cores associated with the respective servers. Certain workloads, while able to run on the allocated compute cores, are less efficiently executed against the compute cores due to the nature of the compute core architecture and / or the algorithms used in these workloads.

[0002] Certain computing environments include hardware accelerators that are able to more efficiently execute these workloads. While using hardware accelerators can more efficiently execute workloads, the decision as to whether to offload a particular workload from a compute core to a hardware accelerator is still inefficient. Accordingly, there is a need for an improved method and system for deciding when a workload that can be offloaded to a hardware accelerator should be offloaded from a compute core. SUMMARY

[0003] One aspect of the present disclosure relates to a method comprising a client application directly submitting, to a hardware accelerator, a command for executing a workload, wherein the command includes an indication of a performance expectation from the hardware accelerator, and wherein the workload is able to be executed by a compute core accessible to the client application or by the hardware accelerator. The method can further comprise, upon receiving a retry response from the hardware accelerator, the client application executing the workload using a compute core accessible to the client application, wherein the hardware accelerator is configured to provide the retry response directly to the client application upon determining that the hardware accelerator is unable to meet the performance expectation.

[0004] In another aspect, the present disclosure relates to a method comprising allowing a client application executing in a user space associated with a virtual computing environment to access an accelerator side to a hardware accelerator capable of executing a workload, the workload being able to be executed either by a compute core accessible to the client application or by the hardware accelerator. The method can further comprise the hardware accelerator providing performance data to the accelerator side.

[0005] The method can further comprise, upon the client application evaluating the performance data obtained from the accelerator side and determining that executing the workload using the compute core will take less time than executing the workload using the hardware accelerator, the client application executing the workload using the compute core accessible to the client application.

[0006] In yet another aspect, the present disclosure relates to a system comprising an accelerator side to allow a plurality of client applications to access one or more of a plurality of shared hardware accelerators, wherein each of the plurality of client applications is capable of executing a workload using a compute core or by using one of the plurality of shared hardware accelerators. The system can further comprise a hardware accelerator among the plurality of shared hardware accelerators configured to receive a command from a client application for executing a workload via a shared bus system coupled to the compute core and the plurality of shared hardware accelerators, wherein the command comprises an indication of a performance expectation from the hardware accelerator, and wherein the workload is capable of being executed by the compute core accessible to the client application or by the hardware accelerator.

[0007] The client application can be configured to execute the workload using the compute core accessible to the client application upon receiving the retry response from the hardware accelerator, wherein the hardware accelerator is configured to provide the retry response directly to the client application upon determining that the hardware accelerator is unable to meet the performance expectation.

[0008] This summary is intended to introduce some concepts, which will be further described below in the detailed description. This summary does not necessarily identify key or essential features of the claimed subject matter or necessarily delineate the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0009] The present disclosure is illustrated schematically and not by limitation with the accompanying drawings in which like reference numerals indicate similar elements. The elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale.

[0010] Figure 1 A block diagram of a system for selectively executing workloads using hardware accelerators is shown, in accordance with one example;

[0011] Figure 2 A block diagram of an example hardware acceleration system for use with the system of Figure 1 ;

[0012] Figure 3 An example tracker for use with the hardware acceleration system of Figure 2 ;

[0013] Figure 4 A system environment for implementing a system for selectively executing workloads using hardware accelerators is shown, in accordance with one example;

[0014] Figure 5 A computing platform that can be used to perform certain methods is shown, in accordance with one example;

[0015] Figure 6 A flowchart of a method according to one example is shown; and

[0016] Figure 7 Another flowchart of a method according to one example is shown. DETAILED DESCRIPTION

[0017] Examples described in the present disclosure relate to selectively executing workloads using hardware accelerators. In certain examples, the methods and systems described herein can be deployed in any virtualized environment, including a single computer (e.g., a single server) or a cluster of computers (e.g., a cluster of servers). In certain embodiments, the methods and systems described herein can be deployed in a cloud computing environment for selectively executing workloads based on performance expectations using hardware accelerators. Cloud computing can refer to a model for enabling on-demand network access to a shared pool of configurable computing resources. For example, cloud computing can be employed in the marketplace to offer on-demand access to the shared pool of configurable computing resources. The shared pool of configurable computing resources can be rapidly provisioned via virtualization and released with low management effort or service provider interaction, and then made available for subsequent scaling. The cloud computing model can be composed of various characteristics such as, for example, on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and the like. The cloud computing model can be employed to provide various services models such as, for example, Hardware as a Service (“HaaS”), Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). The cloud computing model can also be deployed using different deployment models such as, for example, a private cloud, community cloud, public cloud, hybrid cloud, and the like.

[0018] In a virtualized computing environment, client applications run using virtual machines (or other such virtual environments) hosted by servers. As an example, resources including compute cores, memory, and input / output (IO) resources are allocated to the virtual machines, allowing them to execute client applications using one or more compute cores associated with the respective server(s). Certain workloads, while capable of running on the allocated compute cores, are less efficiently executed against the compute cores due to the nature of the compute core architecture and / or the algorithms used in these workloads. Examples of these workloads include padding, copying, cyclic redundancy check (CRC) generation, compression, decompression, encryption, decryption, and the like.

[0019] Certain virtualized computing environments include hardware accelerators that can more efficiently execute these workloads. In such computing environments, many guest applications can execute using virtual machines. Depending on the subscription of the guest (other arrangements), the guest applications can be limited in how many compute cores they can access at a given time. The guest applications can wish to offload certain workloads that, when executed using hardware accelerators, can use less power, execute with higher efficiency, or execute with higher performance, while compute cores are freed up to execute other workloads. While such workloads can be executed more efficiently using hardware accelerators, the decision as to whether to offload a particular workload from a compute core to a hardware accelerator can still be inefficient. In a virtualized system environment, guest applications share resources, including compute cores, memory, and hardware accelerators, and the guest applications can either execute workloads using compute cores allocated to them or offload workloads to accelerators.

[0020] The sharing of resources, including hardware accelerators, can cause delays in the execution of workloads by hardware accelerators. As an example, a guest application can issue work to a hardware accelerator that can be busy, such that the time to complete the work is longer than the time to complete the work using a compute core allocated to the guest application. As an example, assume that a guest application wants to offload a memory fill operation, the guest application can offload the work without realizing that it would have been better not to offload the memory fill operation. As another example, a guest application can be preparing to initialize a large memory array, and instead of using a compute core to initialize the memory array, the guest application can offload the work to a different hardware accelerator. The offload operation can execute in a reasonable amount of time, so the decision to offload can prove to be a good decision. Regardless, the guest application often makes inefficient choices in using hardware accelerators if there is no appropriate mechanism for the guest application to specify performance expectations for workloads executable by the hardware accelerator and to know whether the performance expectations will be met.

[0021] Figure 1A block diagram of a system 100 for selectively executing workloads using hardware accelerators is shown, according to one example. The system 100 has example components associated with a virtualized execution environment, including components for user space, kernel space, and hardware. The system 100 allows client applications (e.g., client application 1 (CA1) 102, client application 2 (CA2) 104, and client application N (CAN) 106) to access an accelerator side 110 and submit workloads directly to hardware accelerators supported by the accelerator side 110. The accelerator side 110 is a portion of the user space that allows access to the client applications, including access to virtual functions associated with physical functions for the hardware accelerators. The accelerator side 110 can be configured to allow any of the client applications to have the ability to access one or more shared hardware accelerators.

[0022] As part of the kernel space, the system 100 also includes accelerator drivers coupled with the accelerator side 110. The system 100 can include one or more such accelerator drivers corresponding to respective hardware accelerators included as part of the system 100. In this example, the system 100 includes accelerator driver 1 112, accelerator driver 2 114, and accelerator driver M 116. The system 100 also includes a bus driver 118 including software drivers corresponding to various bus types, including Peripheral Component Interconnect Express (PCIe), Compute Express Link (CXL), or other bus types. The system 100 also includes a host operating system (OS) 120 capable of communicating with various components of the system 100, including the accelerator side 110 and the bus driver 118. The host OS 120 can be any operating system capable of supporting a virtualized environment. In some examples, functionality associated with the host OS 120 can be shared or included as part of a hypervisor (not shown).

[0023] With continued reference to Figure 1As part of the hardware, system 100 includes a bus system 130. Bus system 130 can be a PCIe bus system, a CXL bus system, or other type of bus system. Bus system 130 is coupled to an input / output (IO) and memory management unit (IOMMU) 140. Bus system 130 is also coupled to a memory system 150, which can also be coupled to IOMMU 140. Memory system 150 can include cache (LI, L2, and / or other system level cache), SRAM, and / or DRAM. IOMMU 140 can provide address translation services, allowing client applications and others to access memory. Memory accesses can be organized in pages, blocks, or other forms.

[0024] Still referring to Figure 1 Bus system 130 is also coupled to compute cores 160. Compute cores 160 can correspond to servers or other compute resources accessible to client applications. As previously noted, in a virtualized environment, compute cores can be shared among many client applications based on the respective client applications’ subscriptions to the compute cores. Bus system 130 is also coupled to hardware accelerators that perform certain workloads that can be offloaded to the hardware accelerators by client applications. In this example, the hardware accelerators include accelerator 1 170, accelerator 2 180, and accelerator M 190. Example hardware accelerators can include Direct Data Flow Accelerators (DSA) from Intel, Quick Assist Technology (QAT) accelerators from Intel, Intel Analytics Accelerators (IAX), or any other applicable hardware accelerator. Offloaded workloads can include padding, copying, cyclic redundancy check (CRC) generation, compression, decompression, encryption, decryption, and the like. Further, offloaded workloads can be sequences that include at least two of the aforementioned workloads. As an example, the sequence can include commands for any of the aforementioned workloads. An example sequence can include the following commands: (1) decrypt; (2) decompress; (3) perform CRC check; (4) compress; (5) encrypt; (6) decrypt; (7) decompress; and (8) perform CRC check. This example sequence involves re-compressing and encrypting data that was previously encrypted with a different key and compressed using a different codec. Other sequences are also available.

[0025] In terms of operation of the system 100, the client applications (e.g., client application 1 (CA1) 102, client application 2 (CA2) 104, and client application N (CAN) 106) can access any hardware acceleration functionality that is provided to them via the accelerator side 110. Any suitable system agent (e.g., the host OS 120 or hypervisor) is configured to allocate virtual functions for a given hardware accelerator to virtual machines or similar entities. This allows the client applications to be able to access virtual functions that correspond to physical functions, which in turn correspond to hardware accelerators. The system agent (e.g., the host OS 120 or hypervisor) also maintains knowledge related to the performance of workloads by the computing core(s) allocated to the client applications. At any given time, based on historical or real-time performance testing data, the host OS 120 is able to ensure that the client applications are aware of the performance that a core will provide if the core is used to execute a workload. As an example, assume that a workload includes a data transfer from one location in memory to another location in memory, and the client application is aware that such an operation can be performed at 20 GB / second. As described further below, a tracker associated with the hardware acceleration system can provide the client application with performance-related information, allowing the client application to decide whether to use the core to execute the workload, or to have the hardware accelerator system execute the workload. Such performance data can also include information about the amount of time required to execute certain workloads using a computing core.

[0026] Each virtual function can only be allocated to a single virtual machine (or another type of guest OS or container) at a time, as the virtual function requires real hardware resources associated with the hardware accelerator. A virtual machine can be allocated multiple virtual functions. The virtual functions are derived from a physical function, which can be a full PCI Express (PCIe) device or another type of device. As an example, one or more PCIe devices can be configured in accordance with the Single-Root Input / Output Virtualization (SR-IOV) specification to provide virtual functions that correspond to physical functions. In addition to the abstraction of physical functions and virtual functions, other abstractions associated with the hardware accelerator can also be provided by the accelerator side 110. The system agent (e.g., the host OS 120 or hypervisor) can configure the specifics of each such allocation, which can involve performance in terms of priority methods and minimum guarantees, among other aspects.

[0027] Once the virtual functions are assigned, the client application is able to directly submit workloads to the hardware accelerators with corresponding physical functions that are supported by the accelerator side 110. In the system 100, the accelerator side 110 is a part of the user space that allows access to the client application, including access to the virtual functions associated with the physical functions for the hardware accelerators. As an example, a client application executing in the user space is able to directly submit offloaded workloads to the hardware accelerators via the data path 174. Similarly, the host OS 120 is able to directly submit offloaded workloads to the hardware accelerators via the data path 172. In one example, the workloads can be submitted using commands that enable memory-mapped I / O operations to be performed directly on the hardware accelerators. In other words, the client application does not need to rely on interrupts or calls to the driver stack in order to submit workloads to the hardware accelerators. Furthermore, the client application does not need to switch out of the context associated with the compute core for executing the client application at a given time. Advantageously, this arrangement reduces the entry cost of offloading workloads to the hardware accelerators by reducing the latency associated with submitting workloads to the accelerators and getting completion results. Furthermore, in a system 100 that includes multiple accelerators, the client application can also issue commands that can be executed by one or more of the accelerators before attempting arbitrary execution using the compute core. As an example, one or more virtual functions can be assigned to a single virtual machine in each of two or more hardware accelerators. The VM can first submit a workload to one of the accelerators with a particular performance expectation. If the hardware accelerator responds with a retry command, the VM can submit the same workload to a different one of the hardware accelerators. The hardware and software associated with the hardware acceleration system 200 can be configured to perform load balancing to determine which of the hardware accelerators should handle the resubmitted workload. Similarly, the same function can be used to initially assign workloads to a selected one of the hardware accelerators to ensure load balancing.

[0028] The client application can submit commands for executing workloads via the data path 174. Such commands will include information related to the client-specific performance expectations for executing the workloads. Such information can be provided as part of a field in a command descriptor for the command. Both posted write commands and non-posted write commands can be used. A posted write command does not require an acknowledgement that the command has been received and has been queued. Instead, an acknowledgement is only sent when the workload processing requested by the posted write command has completed. A non-posted write command allows a response to be returned, which can include a retry response.

[0029] In addition, in some instances, additional information, such as the busy level of a hardware accelerator, can also be provided as part of the response. For example, such information can be provided as part of the completion log once the workload has been executed. Client applications can use the received "busyness" information to modify their performance expectations from a particular hardware accelerator. "Busyness" information can be provided regardless of whether the workload is accepted, rejected, successfully completed after acceptance, or failed. As another example, a hardware accelerator can let a client application know that it has accepted the workload and expects the execution of the workload to complete in a certain amount of time. The client application can use this information to further determine whether to send a request for execution to that hardware accelerator or another hardware accelerator. The term "retry response" as used herein refers to any type of response that allows the client application to determine whether to use compute cores accessible to the client application to execute the workload, or to have the hardware accelerator execute the commands submitted for the workload. Client applications can also use other commands, including those that allow atomic command submissions, to request workload processing from hardware accelerators in the system. Although Figure 1 A system 100 is shown that includes a certain number of components (arranged in a specific manner), but system 100 may include more or fewer components arranged in a different manner. As an example, system 100 may include more or fewer hardware accelerators. As another example, memory management and I / O management tasks may be performed by components other than integrated components (such as...). Figure 1 Execution of different components other than IOMMU 140.

[0030] As another example, although Figure 1 A virtual computing environment with multiple client applications and multiple hardware accelerators is described, but even in a scenario with only one hardware accelerator and only one queue for that accelerator, the methods disclosed herein can be executed. In such a scenario, a simpler performance measurement system than that described later can be used. As an example, a heartbeat signal from a single hardware accelerator might be sufficient as a basis for providing a response to client applications that workload execution is in progress. Therefore, in this scenario, the performance expectation set by the client application could simply be whether the hardware accelerator is making progress on the existing workload. Alternatively, the performance expectation set by the client application could be whether the hardware accelerator has failed to make progress on the execution of existing workloads in the queue for that hardware accelerator.

[0031] Figure 2 It shows the relationship with Figure 1 A block diagram of an example hardware acceleration system (HAS) 200 used in conjunction with system 100. HAS 200 includes a performance measurement block 210 coupled to a command queue 220. Performance measurement block 210 also... Figure 1The IOMMU 140 is coupled to it. Furthermore, the performance measurement block 210 is coupled to the accelerator 240, which can be the aforementioned... Figure 1 Any of the hardware accelerators described. Each hardware accelerator may have a dedicated performance measurement block, or such a block may be shared among hardware accelerators. Command queue 220 may be transmitted via... Figure 1 Data path 174 from Figure 1 The client application receives commands. Command queue 220 can also be accessed via data path 172 from... Figure 1 The host OS 120 receives commands. In one example, command queue 220 can be implemented using a first-in, first-out (FIFO) buffer. Command queue 220 can also be implemented via data path 172 from... Figure 1 The host OS 120 receives commands.

[0032] Command queue 220 may include shared queues for specific hardware accelerators. Some command queues within the command queue may be dedicated queues for specific hardware accelerators. Some command queues may be configured to handle high-priority workloads, while others may be configured to handle regular-priority workloads. Some command queues may be designed for larger workloads, and some may be designed for smaller workloads. This arrangement can help reduce completion latency associated with workloads. Command queues may also be configured such that there are no command queues shared between different types of accelerators. As an example, a compressor accelerator may not share any command queues with a decryption accelerator. An accumulator associated with each command queue may include the size of the corresponding command queue. The accumulator may decrement as commands are submitted for execution. The accumulator may increment as additional commands arrive. At any given time, the accumulator for each command queue will have a run count of the number of commands still in the command queue.

[0033] As previously mentioned, client applications can access virtual functions corresponding to physical functions, which in turn correspond to hardware accelerators. In one example, the HAS 200 supports an input command queue that uses atomic 64-bit writes to provide command descriptors. Command commits can be atomic because atomic command commits are performed as a transaction through the interface, and no other commands can be executed during the execution of that atomic commit. Therefore, the 64-bit write atomic commit is executed as a whole before other read / write operations can be executed using the resources allocated to it. Each command queue operates as an independent FIFO operation, processing its own command descriptors.

[0034] In one example, each hardware accelerator is assumed to have one physical function per device, up to 32 virtual functions per device, up to four command queues per PF / VF, and up to 4000 queue entries per device. In practice, different numbers of physical functions / virtual functions per device can be configured. Each PF, VF, and queue can be identified by a unique identifier. Although Figure 2 not shown in FIG. 3, the device driver can partition the resources according to the VFs, and the function driver can partition the resources for the queues that each VF wants to support. Thus, the device driver can set up and configure the respective hardware accelerator as a whole, and the function driver can handle the configuration, attributes, and quality of service settings for a given VF side. As an example, the device driver can store the base entry number and total entry number per PV / VF for the command queue memory allocation. The device driver can also allocate the number of available command queue tracking resources per VF. The function driver can read the configuration registers to determine its allocation of queue trackers and queue memory. The function driver can also program the number of command queues that it wants to use, and allocate a base address and size for each queue in its allocated command queue memory space.

[0035] The HAS 200 also includes a command processor block 230. In this example, the command processor block 230 includes command processors (CPs) 232, 234, 236, and 238. The HAS 200 also includes a workload processor block 240. In this example, the workload processor block 240 includes workload processors (WPs) 242, 244, 246, and 248. The HAS 200 also includes an address translation service 250. Any of the command processors can access the address translation service (ATS) 250 as well as any of the workload processors in the workload processor block 240. Thus, any of the command processors initiates a virtual address to physical address translation, takes the result of the translation, and then invokes any of the workload processors for execution of the workload. In one example, each workload has a size of 4 kilobytes (KB). The workload size can be chosen to have the same granularity as supported by the host OS for address translation purposes. Each workload processor can be considered to have at least a function associated with a DMA engine in that it can independently initiate a memory read, perform an acceleration function if it needs more than just a copy or move, and then perform a memory write as needed. Although Figure 2The HAS 200 is shown to include a certain number of components arranged in a particular way, but the HAS 200 can include more or fewer components arranged in different ways. As an example, the HAS 200 can include more or fewer command queues, more or fewer command processors, and more or fewer workload processors. As another example, memory management and I / O management tasks can be performed by different components than the integrated components such as the IOMMU 140 of Figure 1 FIG. 1.

[0036] The performance measurement block 210 includes a tracker for tracking performance. There is a separate and independent tracker for performance tracking for each client application and each command queue. Thus, the tracker can be configured to track absolute performance (e.g., N megabytes (MB) per second) or relative performance (e.g., percentage of bandwidth) of the transaction bus associated with the accelerator. Broadly, the tracker has an input signal that will be high for every clock (or a certain number of clocks) that the tracked condition occurs. For the lowest performance, the condition can be when a workload unit is executed. For the highest performance (e.g., maximum bandwidth), the condition can be a write or read data strobe. The tracker has another input signal that will be high for every clock (or a certain number of clocks) that the referenced tracked condition occurs. For absolute performance, this input signal is always 1 when counting the system clock, or is a pulse train that tracks a divided down version of the clock. For relative performance, this input signal can correspond to the time when any transaction occurs, not just the tracked transaction.

[0037] Every time the tracked condition occurs, a programmed value is used to increment the tracking level. Every time the referenced untracked condition occurs (for relative tracking) or the coarse clock ticks (for absolute tracking), another programmed value is used to decrement the tracking level. If the tracked item is occurring at the expected rate, the tracker level will remain at the same level. If the tracked item is occurring at a higher rate than the expected rate, the tracker level will rise. If the tracked item is occurring at a lower rate than the expected rate, the tracker level will fall. Clamping can be used to avoid overflow and underflow cases for these increment / decrement operations. In addition, the tracker supports programmable duty cycle asymmetry (indicating burstiness of the work), offset tracking (indicating how far off the specified rate the tracking is), and assertion levels (indicating when to raise an alarm about deviating from the specified rate).

[0038] The same tracker design can be used to track either the minimum rate or the maximum rate. An input configuration pin (strap) can be used to select between the two modes of operation. The output signal of the tracker indicates whether the tracker level has fallen below the minimum level (when configured to track the minimum rate) or risen above the maximum level (when configured to track the maximum rate). This level is determined by a configuration register. As an example, a 16-bit value for this level and the tracker precision can be used. After reset, the tracker can be reset to a mid-range, or can be set to a programmed level. The total dynamic range of the tracker level is determined by the tracker level precision and the increment / decrement level. The longer the time period for which average performance is desired, the smaller the increment level should be to support a larger dynamic range before clamping.

[0039] Figure 3 An example tracker 300 is shown for use with a hardware acceleration system (HAS) 200 of Figure 1 In this example, the tracker 300 includes a rate counter 310, a clock divider 320, and a comparator 330. In addition, the tracker 300 also includes a multiplexer 320 that allows either the clock signal or the divided clock signal to be used with the rate counter 310. The rate counter 310 can be a 16-bit counter that can be reset to the same value as the threshold. Configuration registers (not shown) can be used to store these values for the tracker 300, which can be programmed depending on the use of the tracker in terms of what conditions (e.g., based on bus transactions, address translation service related transactions, memory usage, or some other aspect) are being tracked.

[0040] Data related to the tracked condition can be obtained from specific resource monitoring corresponding to the architecture associated with the compute core. As an example, the compute core can be an ARM core or an x86 compatible core. If the compute core is an ARM core, then ARM architecture adjuncts related to resource monitoring, such as memory partitioning and monitoring (MPAM), can provide data related to the tracked condition. As an example, MPAM can provide data related to bandwidth consumption associated with transactions on a bus (e.g., bus system 130 of Figure 1 If the compute core is an x86 core, then Intel architecture related aspects for resource monitoring can provide data related to the tracked condition. As an example, memory bandwidth monitoring (MBM) provided by Intel can be used to track data related to bandwidth consumption associated with transactions on a bus (e.g., bus system 130 of Figure 1 Other more general resource monitoring hardware / software systems can be used to collect data related to the condition tracked by each tracker (e.g., tracker 300).

[0041] The US INCREMENT signal corresponds to a signal indicating the condition being tracked over time. If an absolute value is being tracked, the THEM DECREMENT signal corresponds to a clock signal (or clock divider signal). Therefore, if an attempt is made to reach a given rate of resource consumption, and the increment and decrement values ​​are configured to track deviations from the given rate, the rate counter value remains approximately the same. Over time, the rate counter value may occasionally increase and then decrease based on activity and time, but the overall level will remain approximately the same. The specific values ​​selected for the increment and decrement related signals define the dynamic range of the tracker 300. As an example, if the tracker is being used to track memory bandwidth, the rate counter 310 can be incremented for each bus transaction (e.g., a 64-byte (64 B) transaction), or a different level of granularity can be selected. As an example, assuming the hardware accelerator is operating on a 4-kilobyte (4 KB) workload, the rate counter 310 is not incremented for each bus transaction, but rather when each 4 KB bus transaction is completed.

[0042] The THEM DECREMENT signal can correspond to a clock signal that indicates the passage of time (e.g., the clock signal CLOCK or a divided version of the clock signal DIVIDED CLOCK). The THEM DECREMENT signal can also correspond to a signal that is being tracked in a relative manner as time passes.

[0043] Comparator 330 is used to compare the output of rate counter 310 with a threshold (e.g., ...). Figure 3 The threshold is compared to the threshold shown in the diagram (THRESHOLD). If tracker 300 is being used to track maximum usage, the threshold can be set to the maximum usage value configured for the resource (e.g., maximum bandwidth usage). The output of comparator 330 indicates whether the usage rate is higher than the maximum rate or some other target rate (e.g., ABOVE THE RATE). On the other hand, if tracker 300 is being used to track minimum usage, the threshold can be set to the minimum usage value configured for the resource (e.g., minimum bandwidth usage). The output of comparator 330 indicates whether the usage rate is lower than the minimum rate or some other target rate (e.g., BELOW THE RATE). Although Figure 3 The tracker 300 is shown to include a certain number of components arranged in a particular manner, but the tracker 300 may include more or fewer components arranged in a different manner.

[0044] In one example, there is a minimum performance tracker for each VF. This tracker counts the 64B work quantums performed by all commands from all queues for a given VF. The work can require read-only operations (e.g., for CRC generation), write-only operations (e.g., for padding), or both (e.g., for copy). For copy commands, the 64B data read-after-write can be configured to represent one work quantum or 2 quantums. In this example, there is a minimum performance tracker for each queue within a VF. This tracker has the same functionality as the VF minimum tracker, but observes traffic only from that specified VF queue.

[0045] In one example, there are two maximum bandwidth trackers, one for read operations and one for write operations. The trackers track the 64B payload quantums for all commands from all queues for a given VF. In addition, in this example, there are two maximum bandwidth trackers for each queue, one for read operations and one for write operations. The trackers track the 64B payload quantums for all commands from that specified queue. In addition, in this example, there is another set of trackers per VF and per queue that track the rate of address translation service (ATS) transactions used by a given resource.

[0046] In summary, the trackers associated with the hardware accelerator system can provide performance-related information to the client application, allowing the client application to decide whether to execute the workload using the core or to let the hardware accelerator system execute the workload. Such performance data can also include information about the amount of time required to execute certain workloads using the compute core. In another example, the hardware accelerator can be provided with an expectation of the maximum time the accelerator will require in command submission itself. Thus, if the command queue is too full and command submission will be delayed due to the queue being full, the client application can execute the workload using the compute core. As mentioned previously, once the hardware accelerator determines that it cannot meet the performance expectation specified by the client application (e.g., how long it will take to complete a given command), the hardware accelerator simply lets the client application know. As an example, if the hardware accelerator thinks that the command will take longer, it simply returns a "retry" status, and the client can execute the command on its own available compute core. Otherwise, the command is accepted and the accelerator executes the work. The trackers can also be configured to continuously track the maximum time the hardware accelerator will require as part of the command submission process itself.

[0047] The response can include additional information beyond just retry, including, for example, how busy the hardware accelerator is. This is because Figure 2 The HAS 200 includes performance measurement functionality integrated into the design, so supporting these mechanisms is possible. As an example,Figure 2 The performance measurement block 210 can perform the calculation by determining the number of command descriptors submitted in the command queue and tracking the performance of the workload processor associated with the hardware accelerator to determine how busy the hardware accelerator is. Alternatively, the performance measurement block 210 can determine whether the hardware accelerator is busy by simply tracking the number of client applications that have workloads that are being processed by the hardware accelerator. If the number of such client applications at a given time is greater than a predetermined threshold, the hardware accelerator can be considered busy.

[0048] Further, based on the hardware implementation, there can be multiple “levels” of support. Instead of an indication of the time it will take for the accelerator to complete the workload, the same mechanism can be used to indicate a queue full criteria (e.g., retry if there are more than N entries already in the queue), a queue workload criteria (e.g., retry if more than N KB of work has been submitted in the queue), an accelerator full criteria (e.g., retry if more than N entries with higher or same priority have been submitted across various queues), an accelerator workload criteria (e.g., retry if more than N KB of work has been submitted across various queues), a workload completion estimate (e.g., how long it will take to complete the workload based on current performance), or some combination of the above.

[0049] In one example, the command can be a non-forwarding write command that allows a response to be returned. The command itself will have a field that will indicate how much time the client application expects the hardware accelerator to take to execute the workload specified by the command. As an example, for an Intel DSA system, the command can be a modified version of the enqueue command (ENQCMD). If the hardware accelerator is unable to complete the workload within the expected time frame, the response will be a retry response. Alternatively, the command will be accepted and executed by the hardware accelerator. Upon receiving the retry response, the client application will not submit the workload to the hardware accelerator system, but will execute the workload using the compute core that is allocated to the client application. As another example, with respect to an ARM-based accelerator, the command can be a modified version of the ST64 command. The client application can also use other commands (including commands that allow for atomic command submission) to request the hardware accelerator in the system to perform workload processing.

[0050] Instead of non-forwarding commands, forwarding commands can be used with a dedicated direct queue submission arrangement. A forward write command does not require an acknowledgement that the command has been received and queued. Instead, an acknowledgement will only be sent when the workload processing requested by the forward write command has been completed. Forward command submission is generally not the preferred method for command submission due to the higher latency. However, in the context of the present disclosure, even forward commands can be used because such commands can beFigure 2 The performance measurement block 210 (or similar arrangement) of the performance measurement block 210 (or similar arrangement) intercepts to make similar determinations as explained above with respect to non- forwarded write commands.

[0051] Figure 4 A system environment 400 for implementing systems and methods is shown, according to one example. In this example, the system environment 400 can correspond to a portion of a data center. As an example, the data center can include a number of rack clusters (including platform hardware such as server nodes, storage nodes, networking nodes, or other types of nodes). The server nodes can be connected to switches to form a network. The network can enable connectivity between various possible combinations of switches. The system environment 400 can include server 1 410 and server N 430. The system environment 400 can also include data center related functions 460, including deployment / monitoring 470, directory / identity services 472, load balancing 474, data center controllers 476 (e.g., software defined network (SDN) controllers and other controllers), and routers / switches 478. Server 1 410 can include host processor(s) 411, host hypervisor 412, memory 413, storage interface controller(s) (SICs) 414, accelerator(s) 415 (e.g., the accelerators described earlier), network interface controller(s) (NICs) 416, and storage disks 417 and 418. Server N 430 can include host processor(s) 431, host hypervisor 432, memory 433, storage interface controller(s) (SICs) 434, accelerator(s) 435 (e.g., the accelerators described earlier), network interface controller(s) (NICs) 436, and storage disks 437 and 438. Server 1 410 can be configured to support virtual machines, including VM1 419, VM2 420, and VMN 421. The virtual machines can also be configured to support applications, such as APP1 422, APP2 423, and APPN 424. Server N 430 can be configured to support virtual machines, including VM1 439, VM2 440, and VMN 441. The virtual machines can also be configured to support applications, such as APP1 442, APP2 443, and APPN 444.

[0052] Continuing with reference to Figure 4 In one example, the system environment 400 can be enabled for multiple tenants using a virtual extensible local area network (VXLAN) framework. Each virtual machine (VM) can be allowed to communicate with VMs in the same VXLAN segment. Each VXLAN segment can be identified by a VXLAN network identifier (VNI). Although Figure 4System environment 400 is shown to include a number of components arranged and coupled in a particular manner, but it can include fewer or more components arranged and coupled in different manners.

[0053] Figure 5 A block diagram of a computing platform 500 (e.g., for implementing certain aspects of the methods and algorithms associated with the present disclosure) is shown in accordance with one example. Computing platform 500 can include processor(s) 502, I / O component(s) 504, memory 506, hardware accelerator(s) 508, sensor(s) 510, database(s) 512, network interface(s) 514, and I / O port(s), which can be interconnected via bus 520. Processor(s) 502 can execute instructions stored in memory 506. I / O component(s) 504 can include user interface devices such as a keyboard, mouse, voice recognition processor, touchscreen, or display. Memory 506 can be any combination of non-volatile or volatile storage (e.g., flash memory, DRAM, SRAM, or other types of memory). Hardware accelerator(s) 508 can include any of the hardware accelerators described previously.

[0054] Sensor(s) 510 can include telemetry or other types of sensors configured to detect and / or receive information (e.g., conditions associated with a device). Sensor(s) 510 can include sensors configured to sense conditions associated with a CPU, memory or other storage components, FPGA, motherboard, baseboard management controller, etc. Database(s) 512 can be used to store data used to generate reports related to workloads executed using cores associated with processor(s) 502 or hardware accelerator(s) 508.

[0055] Network interface(s) 514 can include communication interfaces such as Ethernet, cellular radio, Bluetooth radio, UWB radio, or other types of wireless or wired communication interfaces. I / O port(s) can include Ethernet ports, InfiniBand ports, optical fiber port(s), or other types of ports. Although Figure 5 Computing platform 500 is shown to include a number of components arranged and coupled in a particular manner, but it can include fewer or more components arranged and coupled in different manners. Furthermore, the functionality associated with computing platform 500 can be distributed according to need.

[0056] Figure 6A flowchart 600 of a method according to one example is shown. Step 610 can include a client application directly submitting a command for execution of a workload to a hardware accelerator, where the command includes an indication of a performance expectation from the hardware accelerator, and where the workload is capable of being executed by a compute core accessible to the client application or by the hardware accelerator. In one example, the method can be performed using one or more components associated with the system 100 of Figure 1 Any of the client applications 102, 104, or 106 can submit a command for execution of a workload directly (e.g., via a path 174) to any of the hardware accelerators (e.g., accelerator 1 170, accelerator 2 180, or accelerator M 190). As previously mentioned, the command can include an indication of a performance expectation from the hardware accelerator. The performance expectation can include an amount of time required for the hardware accelerator to execute the workload. Alternatively or additionally, the performance expectation can include an expectation of a maximum time required by the hardware accelerator for the command submission itself. Figure 1 Figure 1 As previously mentioned, the command can include an indication of a performance expectation from the hardware accelerator. The performance expectation can include an amount of time required for the hardware accelerator to execute the workload. Alternatively or additionally, the performance expectation can include an expectation of a maximum time required by the hardware accelerator for the command submission itself.

[0057] The hardware accelerator can include a command queue, and the performance expectation can include a fullness criteria or a workload criteria associated with one or more of the command queues. The hardware accelerator can allow the client application to access a virtual function for at least one physical function associated with the hardware accelerator, and the performance expectation can be a fullness criteria or a workload criteria associated with one or more of the virtual functions. The performance expectation can also be a fullness criteria or a workload criteria associated with the hardware accelerator itself.

[0058] ​Step 620 may include, upon receiving a retry response from the hardware accelerator, the client application executing the workload using compute cores accessible to the client application, wherein the hardware accelerator is configured to directly provide a retry response to the client application after determining that the hardware accelerator cannot meet performance expectations. As previously described, the command may be a non-forwarded write command that allows the response to return. The command itself will have a field indicating the time the client application expects the hardware accelerator to execute the workload specified by the command. As an example, for an Intel DSA system, the command may be a modified version of an enqueue command (ENQCMD). If the hardware accelerator cannot complete the workload within the expected timeframe, the response will be a retry response. Alternatively, the command will be accepted by the hardware accelerator and will be executed. Upon receiving the retry response, the client application will not submit the workload to the hardware accelerator system, but will instead use the compute cores allocated to the client application to execute the workload. Advantageously, this method reduces the entry cost for offloading workloads to the hardware accelerator by reducing the latency associated with submitting workloads to the accelerator and obtaining completion results. As an example, in systems where a client application needs to read (or otherwise obtain) performance data before deciding to offload a workload, there is latency associated with waiting for a response. Using the method described above, the client application can submit a workload, and the client application will only receive a "retry" response if the workload cannot be executed by the hardware accelerator within the performance-related criteria; otherwise, the workload will simply be executed by the hardware accelerator. Therefore, in many instances, the latency associated with offloading workloads to hardware accelerators is advantageously further reduced. Even smaller workloads can be offloaded due to the reduced overhead (e.g., latency) associated with these methods and systems.

[0059] Figure 7 A flowchart 700 according to an example method is shown. Step 710 may include allowing a client application executing in user space associated with a virtual computing environment to access an accelerator side of a hardware accelerator capable of executing compute cores accessible to the client application or workloads executed by the hardware accelerator. Figure 1 As shown, the accelerator end 110 can be configured to enable client applications (e.g., Figure 1 Client applications 1 (102), 2 (104), and N (106) can access the accelerator.

[0060] Step 720 may include the hardware accelerator providing performance data to the accelerator. Figure 1Any of the illustrated hardware accelerators can provide performance data to the accelerator side. This performance data can be various types of performance related data described earlier in that it can help the client application determine whether to offload the workload to the hardware accelerator or execute the workload using the compute cores accessible to the client application.

[0061] Step 730 can include the client application executing the workload using the compute cores accessible to the client application after the client application evaluates the performance data obtained from the accelerator side and determines that executing the workload using the compute cores will be better than executing the workload using the hardware accelerator. As described earlier, the client application can make the evaluation by comparing performance related data for the compute cores accessible to it with the performance data provided by the hardware accelerator. As an example, the client application can determine that the time required to execute the workload using the compute cores accessible to it is less than the time required to execute the workload using the hardware accelerator. Advantageously, this approach also reduces the entry cost for offloading the workload to the hardware accelerator by reducing the latency associated with submitting the workload to the accelerator and obtaining the completed result.

[0062] In summary, the present disclosure is directed to a method that includes a client application directly submitting a command to a hardware accelerator for execution of a workload, where the command includes an indication of a performance expectation from the hardware accelerator, and where the workload can be executed by a compute core accessible to the client application or by the hardware accelerator. The method can also include the client application executing the workload using the compute core accessible to the client application upon receiving a retry response from the hardware accelerator, where the hardware accelerator is configured to directly provide the retry response to the client application upon determining that the hardware accelerator is unable to meet the performance expectation.

[0063] The method can also include the hardware accelerator executing the workload upon determining that the hardware accelerator is able to meet the performance expectation, rather than providing a retry response to the client application. The performance expectation can include an amount of time required for the hardware accelerator to execute the workload. The performance expectation can include a maximum time required for the hardware accelerator to commit to the command.

[0064] The hardware accelerator includes a command queue, and the performance expectation can include a fullness criteria or a workload criteria associated with one or more of the command queues. The hardware accelerator allows the client application to access a virtual function for at least one physical function associated with the hardware accelerator, and the performance expectation can include a fullness criteria or a workload criteria associated with one or more of the virtual functions.

[0065] The performance expectation can include a fullness criteria or a workload criteria associated with the hardware accelerator. The workload can include a copy workload, a fill workload, an encryption workload, a decryption workload, a compression workload, a decompression workload, a cyclic redundancy check (CRC) generation workload, or a sequence including at least two of the aforementioned workloads.

[0066] In another aspect, the disclosure relates to a method that includes allowing a client application executing in a user space associated with a virtual computing environment to access an accelerator side for a hardware accelerator capable of executing a compute core accessible by the client application or a workload executed by the hardware accelerator. The method can also include the hardware accelerator providing performance data to the accelerator side.

[0067] The method can also include, after the client application evaluates the performance data obtained from the accelerator side and determines that executing the workload using the compute core will take less time than executing the workload using the hardware accelerator, the client application executing the workload using the compute core accessible by the client application.

[0068] The method can also include, after the client application evaluates the performance data obtained from the accelerator side and determines that executing the workload using the compute core will be worse than executing the workload using the hardware accelerator, the hardware accelerator executing the workload. The accelerator side can be located in the user space and can be configured to provide access to virtual functions and physical functions associated with a plurality of hardware accelerators.

[0069] The workload can include a copy workload, a fill workload, an encryption workload, a decryption workload, a compression workload, a decompression workload, a cyclic redundancy check (CRC) generation workload, or a sequence including at least two of the aforementioned workloads.

[0070] In yet another aspect, the disclosure relates to a system that includes an accelerator side to allow a plurality of client applications to access one or more of a plurality of shared hardware accelerators, where each of the plurality of client applications can execute a workload using a compute core or by using one of the plurality of shared hardware accelerators. The system can also include a hardware accelerator among the plurality of shared hardware accelerators configured to receive a command from a client application to execute a workload via a shared bus system coupled to the compute core and the plurality of shared hardware accelerators, where the command includes an indication of a performance expectation from the hardware accelerator, and where the workload can be executed by the compute core accessible by the client application or by the hardware accelerator.

[0071] The client application can be configured to execute the workload using a computing core accessible to the client application upon receiving the retry response from the hardware accelerator, where the hardware accelerator is configured to provide the retry response directly to the client application upon determining that the hardware accelerator is unable to meet the performance expectation.

[0072] The hardware accelerator can be configured to execute the workload upon determining that the hardware accelerator is able to meet the performance expectation, rather than providing the retry response to the client application. The performance expectation can include an amount of time required for the hardware accelerator to execute the workload. The performance expectation can include a maximum time required for the hardware accelerator to commit to the command itself.

[0073] The hardware accelerator allows the client application to access a virtual function for at least one physical function associated with the hardware accelerator, and the performance expectation can include a fullness criteria or a workload criteria associated with one or more of the virtual functions. The hardware accelerator includes command queues, and the performance expectation can include a fullness criteria or a workload criteria associated with one or more of the command queues. The performance expectation can include a fullness criteria or a workload criteria associated with the hardware accelerator.

[0074] Each shared hardware accelerator of the plurality of shared hardware accelerators includes a plurality of command queues. Each command queue of the plurality of command queues can have an associated tracker to track at least one performance criteria associated with the respective queue. The workload can include a copy workload, a fill workload, an encryption workload, a decryption workload, a compression workload, a decompression workload, a cyclic redundancy check (CRC) generation workload, or a sequence including at least two of the aforementioned workloads.

[0075] It should be understood that the methods, modules, and components described herein are merely exemplary. Alternatively, or additionally, the functions described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc. In an abstract, but still definite sense, any arrangement of components to accomplish the same functionality is effectively "associated" such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as "associated with" each other such that the desired functionality is achieved, irrespective of architectures or intermediate components. Likewise, any two components so associated can also be viewed as being "operably connected," or "coupled," to each other to achieve the desired functionality.

[0076] The functionality associated with some of the examples described in this disclosure can also include instructions stored in non-transitory media. As used in this document, the term “non-transitory media” is intended to refer to any media that stores data and / or instructions that cause a machine to operate in a specific manner. Example non-transitory media include non-volatile media and / or volatile media. Non-volatile media includes, for example, a hard disk, a solid state drive, a magnetic disk or tape, an optical disk or tape, flash memory, EPROM, NVRAM, PRAM, or other such media, or a networked version of any of such media. Volatile media includes, for example, dynamic memory, such as DRAM, SRAM, a cache or other such media. Non-transitory media is distinct from, but can be used in combination with, transmission media. Transmission media is used to transmit data and / or instructions to and / or from a machine. Example transmission media include coaxial cables, fiber optic cables, copper wires, and wireless media, such as radio waves.

[0077] Furthermore, those skilled in the art will recognize that boundaries between the functionality of the above described operations merely illustrative. The functionality of multiple operations can be combined into a single operation, and / or the functionality of a single operation can be distributed in additional operations. Moreover, alternative embodiments can include multiple instances of a particular operation, and the order of operations can be altered in various other embodiments.

[0078] Although specific examples have been provided in the present disclosure, various modifications and changes can be made without departing from the scope of the present disclosure, which is set forth in the following claims. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure. Any benefits, advantages, or solutions to problems that are described herein as associated with specific examples can not be intended to be critical, essential, or mandatory for all claims.

[0079] Furthermore, as used herein the term “a” and “an” are defined as one or more than one. Also, the use of the interchanging of the phrases “at least one” and “one or more” in the claims does not imply that any particular claim element is limited to only one of the recited elements, even though the phrase “a” or “an” is used in the claim element. The same applies to the use of the phrases “at least one of” and “one or more of” in the specification.

[0080] Unless otherwise stated, the terms such as “first” and “second” are used to arbitrarily distinguish one element from another element, for example, without necessarily requiring priority or order. Therefore, the above terms are not intended to necessarily indicate a particular priority or order.

Claims

1. A method comprising: a client application (102, 104, or 106) submitting a command directly to a hardware accelerator (170, 180, or 190) for execution of a workload, wherein the command includes an indication of a performance expectation from the hardware accelerator (170, 180, or 190), and wherein the workload is capable of being executed by a compute core (160) accessible to the client application (102, 104, or 106) or by the hardware accelerator (170, 180, or 190) (step 610); and upon receiving a retry response from the hardware accelerator (170, 180, or 190), the client application (102, 104, or 106) executing the workload using the compute core (160) accessible to the client application (102, 104, or 106), wherein the hardware accelerator (170, 180, or 190) is configured to provide the retry response directly to the client application (102, 104, or 106) upon determining that the hardware accelerator (170, 180, or 190) is unable to meet the performance expectation (step 620).

2. The method of claim 1, further comprising: upon determining that the hardware accelerator is capable of meeting the performance expectation, the hardware accelerator executing the workload instead of providing the retry response to the client application.

3. The method of claim 1, wherein the performance expectation includes an amount of time required for the hardware accelerator to execute the workload.

4. The method of claim 1, wherein the performance expectation includes a maximum time required for the hardware accelerator for command submission itself.

5. The method of claim 1, wherein the hardware accelerator includes a command queue, and wherein the performance expectation includes a fullness criteria or a workload criteria associated with one or more of the command queues.

6. The method of claim 1, wherein the hardware accelerator allows a client application to access a virtual function for at least one physical function associated with the hardware accelerator, and wherein the performance expectation includes a fullness criteria or a workload criteria associated with one or more of the virtual functions.

7. The method of claim 1, wherein the performance expectation includes a fullness criteria or a workload criteria associated with the hardware accelerator.

8. The method of claim 1, wherein the workload includes a copy workload, a fill workload, an encryption workload, a decryption workload, a compression workload, a decompression workload, a cyclic redundancy check (CRC) generation workload, or a sequence including at least two of the aforementioned workloads.

9. A system (100) comprising: an accelerator side (110) to allow a plurality of client applications (102, 104, 106) to access one or more shared hardware accelerators (170, 180, or 190) from among a plurality of shared hardware accelerators (170, 180, or 190), wherein each of the plurality of client applications (102, 104, 106) is capable of executing a workload using a compute core (160) or by using one of the plurality of shared hardware accelerators (170, 180, or 190); a hardware accelerator (170, 180, or 190) from among the plurality of shared hardware accelerators (170, 180, or 190) configured to receive a command from a client application (102, 104, 106) for executing a workload via a shared bus system coupled to the compute core (160) and the plurality of shared hardware accelerators (170, 180, or 190), wherein the command includes an indication of a performance expectation from the hardware accelerator (170, 180, or 190), and wherein the workload is capable of being executed by the compute core (160) accessible to the client application (102, 104, or 106) or by the hardware accelerator (170, 180, or 190); and the client application (102, 104, or 106) configured to execute the workload using the compute core (160) accessible to the client application (102, 104, or 106) upon receiving a retry response from the hardware accelerator (170, 180, or 190), wherein the hardware accelerator (170, 180, or 190) is configured to provide the retry response directly to the client application (102, 104, or 106) upon determining that the hardware accelerator (170, 180, or 190) is unable to meet the performance expectation.

10. The system of claim 9, wherein the hardware accelerator is configured to execute the workload upon determining that the hardware accelerator is capable of meeting the performance expectation, instead of providing the retry response to the client application.

11. The system of claim 9, wherein the performance expectation includes an amount of time required for the hardware accelerator to execute the workload.

12. The system of claim 9, wherein the performance expectation includes a maximum time required for the hardware accelerator for command submission itself.

13. The system of claim 9, wherein the hardware accelerator allows a client application to access a virtual function for at least one physical function associated with the hardware accelerator, and wherein the performance expectation includes a fullness criteria or a workload criteria associated with one or more of the virtual functions.

14. The system of claim 9, wherein the hardware accelerator includes a command queue, and wherein the performance expectation includes a fullness criteria or a workload criteria associated with one or more of the command queues.

15. The system of claim 9, wherein the performance expectation comprises a fullness criterion or a workload criterion associated with the hardware accelerator.

Images