Reducing Latency in Highly Scalable HPC Applications via Accelerator-Resident Runtime Management

Accelerator-resident managers manage kernel execution and synchronization to reduce latency in CPU-APU communication, improving performance scaling and efficiency in HPC applications by dynamically handling dependencies and load balancing.

JP2024541294APending Publication Date: 2024-11-08ADVANCED MICRO DEVICES INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024526933
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-11
Filing Date
2022-10-26
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

High latency in CPU-APU communication, particularly in strongly-scaled High Performance Computing (HPC) applications, is exacerbated by kernel synchronization and dependency changes during runtime, which can rival or exceed kernel execution time.

Method used

Implementing accelerator-resident managers to manage kernel startup and synchronization, decoupling the accelerator from CPU control, and utilizing a graph-based approach to dynamically assign and execute kernels on APUs based on dependencies, with asynchronous updates and load balancing.

Benefits of technology

Reduces latency by optimizing CPU-APU communication and improving performance scaling through efficient management of kernel execution and dependency handling, enhancing HPC application efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541294000001_ABST
    Figure 2024541294000001_ABST
Patent Text Reader

Abstract

A method and system for run-time management by an accelerator residency manager is provided. The technique includes a manager receiving a representation of an application's processing flow including multiple kernels and their respective dependencies. The manager then assigns the multiple kernels to one or more APUs that it manages, launches the multiple kernels on their assigned APUs, and executes them iteratively according to their respective dependencies.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Patent Application No. 17 / 454,607, filed November 11, 2021, the contents of which are incorporated herein by reference. [Background technology]

[0002] Accelerated Processing Units (APUs) are well-designed to efficiently execute thousands of processes (computational tasks or kernels) in parallel. A typical application may involve processes that must be executed sequentially and processes that can be parallelized. Thus, an application software architecture may include a sequential portion that may be designed to run on a Central Processing Unit (CPU) and a parallelizable portion that may be designed to run on an APU or an accelerator including a cluster of APUs to achieve the high performance afforded by the cluster. However, significant latency is introduced by the CPU-APU communication as the processes running on the CPU manage the work (such as synchronizing and launching kernels) that is performed on the APU. Such latency is particularly important for strongly-scaled High Performance Computing (HPC) applications where a myriad of kernels with small workloads are managed by the CPU. In such cases, the time spent by the CPU in synchronizing and launching kernels may rival or exceed the execution time of the kernels themselves. Furthermore, changes in dependencies between kernels that occur during runtime can further contribute to the latency associated with CPU-APU communication by requiring the CPU to suspend runtime operations to adjust synchronization schemes. Techniques are needed to reduce the latency associated with CPU-APU communication.

[0003] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0004] [Figure 1A] FIG. 1 is a block diagram of an exemplary device, upon which one or more features of the present disclosure may be implemented. [Figure 1B] FIG. 1B is a block diagram of an example system illustrating an acceleration system usable by the device of FIG. 1A, based on which one or more features of the present disclosure may be implemented. [Diagram 2] FIG. 1 is a block diagram of an example system illustrating centralized management of application runtime, according to which one or more features of the present disclosure may be implemented. [Diagram 3] 1 is an exemplary graph based on which one or more features of the present disclosure may be implemented. [Figure 4] FIG. 1 is a block diagram of an example system illustrating distributed management of application runtime, according to which one or more features of the present disclosure may be implemented. [Diagram 5] FIG. 1 is a block diagram of an example system illustrating distributed management of application runtime across network boundaries, according to which one or more features of the present disclosure may be implemented. [Figure 6] 1 is a flowchart of an example method for accelerator resident runtime management, according to which one or more features of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] HPC applications often benefit from parallelized computations that can run independently on many data elements. Performance scaling can be achieved by systems that include multiple clusters of APUs on which kernels can run in parallel, with each kernel performing computations on one or more data elements. In such a regime, latency can be increased by various components of the system, e.g., the network-interface controller (NIC), the CPU, or the APU. Communication between these components is often a key performance limitation.

[0006] Disclosed herein are systems and methods for reducing latency contributed by communication between a CPU and an accelerator (e.g., one or more clusters of APUs). Such communication is necessitated by the role of the CPU in managing the execution of kernels on the accelerator. Described herein is a technique in which management of application execution flow is performed by an accelerator resident manager, i.e., management functions are performed by a module resident on each accelerator. The disclosed accelerator resident manager controls, for example, kernel launch and synchronization, and decouples the accelerator from control of the CPU during run-time.

[0007] A method for run-time management by an accelerator residency manager is disclosed herein that includes receiving, by the manager, a representation of an application's processing flow including multiple kernels and their respective dependencies, allocating, by the manager, the multiple kernels to one or more APUs managed by the manager, and launching, by the manager, the multiple kernels to execute in iterations on their assigned APUs according to their respective dependencies.

[0008] A system for run-time management by an accelerator residency manager is also disclosed. The system includes at least one processor and a memory that stores instructions that, when executed by the at least one processor, cause the system to: receive, by the manager, a representation of a processing flow of an application including multiple kernels and their respective dependencies; assign, by the manager, the multiple kernels to one or more APUs managed by the manager; and launch, by the manager, the multiple kernels to execute in iterations on their assigned APUs according to their respective dependencies.

[0009] Further disclosed herein is a non-transitory computer-readable storage medium comprising instructions executable by at least one processor to perform a method for run-time management by an accelerator residency manager, the method including receiving, by the manager, a representation of a processing flow of an application including a plurality of kernels and their respective dependencies, allocating, by the manager, the plurality of kernels to one or more APUs managed by the manager, and launching, by the manager, the plurality of kernels to execute in iterations on their assigned APUs according to their respective dependencies.

[0010] FIG. 1A is a block diagram of an exemplary device 100A, based on which one or more features of the present disclosure may be implemented. The device 100A may be, for example, a computer, a gaming device, a handheld device, a set-top box, a television, a mobile phone, or a tablet computer. The device 100A includes a processor 102, an APU 106, a memory 104, a storage 116, an input device 108, and an output device 110. The device 100A may also include an input driver 112 and an output driver 114. In one aspect, the device 100A may include additional components not shown in FIG. 1A.

[0011] The processor 102 may include a CPU or one or more cores of a CPU. The APU 106 may represent a highly parallel processing unit, a graphics processing unit (GPU), or a combination thereof. The processor 102 and the APU 106 may be located on the same die or on separate dies. The memory 104 may be located on the same die as the processor 102 or may be located separately from the processor 102. The memory 104 may include volatile or non-volatile memory (e.g., random access memory (RAM), dynamic random access memory (DRAM), cache, or a combination thereof).

[0012] Storage 116 may include fixed or removable storage (e.g., hard disk drive, solid state drive, optical disk, flash drive). Input device 108 may represent one or more input devices such as a keyboard, keypad, touch screen, touch pad, detector, microphone, accelerometer, gyroscope, biometric scanner, or network connection (e.g., wireless local area network card for receiving wireless IEEE 802 signals). Output device 110 may represent one or more output devices such as a display, speaker, printer, haptic feedback device, one or more lights, antennas, or network connection (e.g., wireless local area network card for transmitting wireless IEEE 802 signals).

[0013] Input driver 112 communicates with processor 102 and input device 108 and facilitates receiving input from input device 108 to processor 102. Output driver 114 communicates with processor 102 and output device 110 and facilitates sending output from processor 102 to output device 110. In one aspect, input driver 112 and output driver 114 are optional components and device 100A can operate in the same manner in the absence of input driver 112 and output driver 114.

[0014] APU 106 may be configured to accept computational and graphics rendering commands from processor 102, process those computational and graphics rendering commands, and / or provide output to a display (output device 110). As described in more detail below, APU 106 may include one or more parallel processing units configured to perform computations according to, for example, a single instruction multiple data (SIMD) paradigm. Thus, although various functions are described herein as being performed by or in conjunction with APU 106, in various alternative examples, functions described as being performed by APU 106 may additionally or alternatively be performed by other computing devices having similar capabilities that are not driven by a host processor (e.g., processor 102) and that may be configured to, for example, provide graphical output to a display. A processing system may be configured to perform the functions described herein regardless of whether the processing system is capable of performing processing tasks according to the SIMD paradigm.

[0015] FIG. 1B is a block diagram of an exemplary system 100B illustrating an acceleration system usable by the device of FIG. 1A, based on which one or more features of the present disclosure may be implemented. FIG. 1B illustrates in further detail the execution of processing tasks on the APU 106. The processor 102 may maintain in the memory 104 one or more modules for execution by the processor 102. The modules include an operating system 120, a kernel mode driver 122, and an application 126. These modules may control various aspects of the operation of the processor 102 and the APU 106. For example, the operating system 120 may provide system calls, i.e., application programming interfaces (APIs), that may be employed by the application 126 to directly interface with hardware. The kernel mode driver 122 may control the operation of the APU 106, for example, by providing an API to the application 126 executing on the processor 102 to access various functions of the APU 106.

[0016] APU 106 can execute commands related to graphics and non-graphics operations, including either parallel or sequential processing, and either sequential or non-sequential processing. APU 106 can be used to perform graphics pipeline operations, such as operations that process pixels and / or geometry calculations (e.g., rendering an image to a display (output device 110)) based on commands received from processor 102. APU 106 can also execute processing operations not related to graphics operations, such as operations related to multi-dimensional data, physics simulations, computational fluid dynamics, or other computational tasks, based on commands received from processor 102.

[0017] APU 106 may include WGPs 132.1-132.M, where each WGP, e.g., 132.1, may have one or more SIMD units, e.g., 138.1.1-138.1.N, capable of executing operations in a parallel manner according to the SIMD paradigm. The SIMD paradigm is one in which multiple processing elements share a single program control flow unit and program counter, thereby allowing the same program to be executed on different data. In one example, each SIMD unit, e.g., 138.1.1, may execute 64 lanes (i.e., threads), each lane executing the same instruction simultaneously with other lanes in the SIMD unit, but executing the instruction on different data. Lanes may be switched off with prediction, such as when not all lanes need to execute a given instruction. Prediction may also be used to execute programs with branching control flows. Specifically, for programs with conditional branches (or other instructions where control flow is based on calculations performed by individual lanes), prediction of lanes corresponding to currently not executed control flow paths and serial execution of different control flow paths allows arbitrary control flow. In one embodiment, each of WGPs 132.1-132.M may have a local cache. In another embodiment, multiple WGPs may share a cache.

[0018] The basic unit of execution within a WGP, e.g., 132.1, is the work item. Typically, each work item represents a single instantiation of a program that may be executed in parallel on a particular lane. Work items may be executed simultaneously as a "wavefront" (or "wave") on a single SIMD, e.g., 138.1.1. One or more waves may be executed in a workgroup, with each wave including a collection of work items designated to execute the same program. A workgroup is executed by executing each of the waves that make up the workgroup. Also, waves may be executed sequentially on a single SIMD unit, or may be executed partially or fully in parallel on different SIMD units 138.1.1-138.1.N. Thus, a wave can be thought of as a collection of work items that may be executed simultaneously on a single SIMD unit, e.g., 138.1.1. Thus, if commands received from processor 102 indicate that a particular program should be parallelized to the extent that the program cannot be run simultaneously on a single SIMD unit, then the program may be divided into waves that may be parallelized on two or more SIMD units (e.g., 138.1.1-138.1.N), serialized on the same SIMD unit (e.g., 138.1.1), or both parallelized and serialized as desired. Scheduler 136 may be configured to perform operations associated with initiating various waves on different WGPs 132.1-132.M and their respective SIMD units.

[0019] The parallelism provided by WGPs 132.1-132.M is well suited for graphics-related operations such as, for example, operations on pixel values ​​(e.g., filter operations), operations on geometric data (e.g., vertex transformations), and other graphics-related operations. For example, an application 126 executing on processor 102 may involve computations performed by APU 106. Application 126 may issue processing commands to APU 106 using an API provided by kernel mode driver 122. The processing commands are then provided to scheduler 136. Scheduler 136 converts the processing commands into computational tasks that are assigned to WGPs 132.1-132.M for parallel execution. For example, scheduler 136 may receive processing commands that include instructions to be performed on data (e.g., 1024 pixels of an image). In response, scheduler 136 may divide the data into groups (e.g., each group containing data required to process 64 pixels) and initiate waves in one or more WGPs, where each wave is associated with a group of data and an instruction to perform on the data. For example, scheduler 136 may initiate 16 waves (e.g., each responsible for processing 64 pixels) to be executed in SIMD 138 of one or more WGPs 132.

[0020] 2 is a block diagram of an example system 200 illustrating centralized management of an application runtime, based on which one or more features of the present disclosure may be implemented. System 200 includes accelerator 210, memory 250, and processor 270. Accelerator 210 may include manager 220 and one or more APUs (e.g., APU 106 of FIG. 1B), such as APU-1 230 and APU-2 240. Processor 270 (e.g., processor 102 of FIG. 1A) may utilize an API exposed by manager 220 to initiate an expression that defines the processing flow of an application to be executed by accelerator 210.

[0021] A representation of an application's processing flow may be represented by a graph 260 (as further described by FIG. 3 ) that defines the application's kernels and dependencies between the kernels. Such a graph 260 may be stored in and accessed from memory 250. When the representation is initiated, manager 220 may assign each kernel to an APU of accelerator 210 and launch the kernels on those assigned APUs to execute in one or more iterations according to the processing flow defined by the representation. For example, if a representation of an application's processing flow is represented by graph 260, when the graph is initiated, manager 220 may partition the graph among the APUs of accelerator 210 such that kernels associated with one graph's partition are assigned to one APU, e.g., APU-1 230, and kernels associated with another graph's partition are assigned to another APU, e.g., APU-2 240. Manager 210 may then execute the application according to the graph by launching the kernels to execute on their assigned APUs according to the dependencies defined by graph 260. Manager 220 may be implemented by dedicated hardware (e.g., an embedded RISC processor, a special purpose APU), by a process running on one of APUs 230, 240, or by a combination of hardware and software.

[0022] 3 illustrates an example graph 300 based on which one or more features of the present disclosure can be implemented. The nodes 310 of the graph 300 represent kernels. Each kernel is, for example, a task that processes one or more data elements stored in the memory 250. Thus, each kernel receives input data and delivers output data (e.g., a processed version of the input data). The edges 320 of the graph 300 represent data dependencies. For example, edge 320.1.2 represents that kernel B 310.2 depends on kernel A 310.1, which means that kernel B 310.2 must operate on data generated by kernel A 310.1 (or the input to kernel B 310.2 is the output of kernel A 310.1), and therefore kernel B 310.2 cannot begin execution until kernel A 310.1 has first completed its own execution.

[0023] Thus, the manager 220 can use the graph 300 to determine in what order the kernels are launched based on the kernel dependencies described by the graph. Additionally, each node is assigned to an APU according to the partition determined by the manager. As shown in FIG. 3, the graph 300 may be partitioned into two sub-graphs 330 and 340 by a graph boundary 350, where the sub-graph 330 on the left side of the graph boundary 350 may be assigned to be executed by the APU-1 230 of FIG. 2, and the sub-graph 340 on the right side of the graph boundary 350 may be assigned to be executed by the APU-2 240 of FIG. 2.

[0024] Thus, in addition to kernels and their assigned APUs, the nodes of the graph, e.g., 310.1-310.8, may have other data entities associated with them that may be useful in managing the application runtime by manager 220. Each node may have one or more mailboxes associated with it (e.g., stored in memory 250). A mailbox at a node may be used by another kernel (on which the kernel associated with the node depends) to leave a message indicating that the other kernel has completed its execution (i.e., a completion message), so that the associated kernel of the node can begin its own execution unless it is still waiting for more other kernels to complete their execution. For example, node 310.7 may have two mailboxes associated with it, one used by kernel B (associated with node 310.2) and the other used by kernel C (associated with node 310.3). Upon completing their respective execution, kernels B and C send messages to that effect to their respective mailboxes of node 310.7. Task G, which is waiting for these mailboxes, may begin its own execution upon receiving these completion messages.

[0025] In one aspect, each node of the graph 300 may be associated with as many mailboxes as the number of dependencies its associated kernel has. When a kernel completes operation on its assigned APU, it may send a completion message to the mailboxes where the kernel is registered. These are mailboxes that are dedicated to that kernel and are located on nodes that branch off from the kernel's node. Once all mailboxes in a node receive messages indicating completion of kernels registered in these mailboxes, the kernel associated with that node may be immediately executed by its assigned APU. Meanwhile, the manager may asynchronously update the graph based on, for example, the status of the graph's mailboxes.

[0026] In one aspect, the graph can dynamically change its structure as dependencies change during execution of an iteration of the application at runtime. For example, the graph may represent kernels designed to solve a physics problem, where each kernel is involved in processing data associated with a particular physical entity, and dependencies between kernels originate from the spatial distance between the corresponding entities. Thus, if a first kernel is designed to process data associated with entity Q and a second kernel is designed to process data associated with entity P, the second kernel may depend on the first kernel as long as entities Q and P are in spatial proximity to each other. However, if entities Q and P move away from each other (as may occur in a simulation of a physical system), the dependency between the first kernel and the second kernel ends. In such a case, to respond to such changes without interrupting runtime operation, the first kernel can send a message to its dedicated mailbox at the node associated with the second kernel indicating that the second kernel is no longer dependent on the first kernel. As a result, the second kernel (as long as it does not depend on other kernels that have not yet completed) can begin its own execution on its assigned APU. Thus, for example, a mailbox at node 310.7 (associated with kernel G) may be registered to receive messages from kernel C (associated with node 310.3). That mailbox may receive a message indicating that kernel C has completed (i.e., a completion message) or a message indicating that a dependency on kernel C no longer exists (i.e., a termination message). Both messages may prompt execution of kernel G. Independent of breaking (or terminating) dependencies directly via mailbox communication, manager 220 may asynchronously update the graph to reflect changes in dependencies based on messages stored in the graph's mailbox.For example, at the end of an iteration, the manager can update the partitioning of the graph (the allocation of kernels to APUs) based on changes in dependencies.

[0027] In addition to resulting in the termination of existing dependencies, as discussed above, changes in dependencies can also create new dependencies. Consider the example discussed above, where the graph represents kernels designed to solve a physical problem, and dependencies between these kernels stem from the spatial distance between the corresponding entities. If a first kernel is designed to process data associated with entity Q and a second kernel is designed to process data associated with entity P, the second kernel may not depend on the first kernel as long as entities Q and P are spatially distant from each other. However, if entities Q and P move closer to each other (as may occur in a simulation of a physical system), a dependency between the first and second kernel may be created. In such a case, to respond to such changes without interrupting runtime operation, the first kernel may create a new mailbox associated with and registered (or dedicated) to the node of the second kernel. Then, when its operation is completed, the first kernel may leave a completion message in its mailbox. Thus, for example, kernel G at node 310.7 is shown to be independent of kernel D. However, when a dependency is created, kernel D can create a new mailbox in node 310.7 for itself. When kernel D completes its operation, it can send a completion message to the new mailbox. Upon receiving the completion message, kernel G can begin its own execution (as long as it is not already waiting to receive completion messages from kernels B and C). As described above, independent of creating new dependencies directly via mailbox communication, the manager 220 can asynchronously update the graph to reflect changes in dependencies based on messages stored in the graph's mailbox. For example, at the end of an iteration, the manager can update the graph's partition (the allocation of kernels to APUs) based on changes in dependencies.

[0028] In another aspect, manager 220 can perform load balancing at the end of each iteration by monitoring the time it takes each APU to complete the kernels assigned to it, and then manager 220 can repartition the graph to improve the load balance across all APUs. Thus, the nodes of the graph, e.g., 310.1-310.8, can have timers associated with them. The timers of the nodes can be set to measure the execution time of the kernels associated with the nodes. Thus, at the end of each iteration of graph 300, manager 220 can repartition the graph to load balance its operations based on an analysis of the timer readings. For example, at the end of an execution iteration, based on an analysis of the timer readings, manager 220 can find that APU-2 240 took more time to execute the kernels associated with its own assigned sub-graph 340 compared to the time APU-1 230 took to execute the kernels associated with its own assigned sub-graph 330. In such a case, manager 220 may decide to load balance the graph by, for example, repartitioning the graph so that node 310.5 becomes part of subgraph 330 and, therefore, kernel E associated with node 310.5 will be executed by APU-1 230 in the next iteration of the graph.

[0029] 4 is a block diagram of an example system 400 illustrating distributed management of application runtime, based on which one or more features of the present disclosure may be implemented. System 400 may include multiple accelerators 410.1-410.M, shared memory 440, and processor 460. Each accelerator, for example 410.1, may include a manager 420.1 and a cluster of APUs 430.1.1-430.1.N. Processor 460 may utilize an API exposed by any manager to initiate a graph 450 that defines the processing flow of an application executed by system 400. The initiated graph 450 may be stored in shared memory 440 accessible by managers 420.1-420.M, clusters of APUs 430.1.1-430.1.N, 430.2.1-430.2.N, 430.M.1-430.MN, and processor 460 of system 400. Once graph 450 is initiated, any manager, e.g., 420.1, may be responsible for partitioning the graph among accelerators 410.1-410.M, resulting in accelerator-level subgraphs. Each accelerator's graph manager, e.g., 420.m, may then proceed to further partition its respective portion of the graph (its respective accelerator-level subgraph) into APU-level subgraphs. Each such APU-level subgraph (or subgraph) is assigned to one APU of accelerator 420.m. Once the partitioning of graph 450 into subgraphs is complete, each manager 420.m may begin with the execution of its own portion of graph 450 in one or more iterations by launching kernels in its respective APU 420.m.1-420.m.N according to dependencies defined by that portion of graph 450. Each manager 420.m may be implemented by dedicated hardware (eg, an embedded RISC processor, a special purpose APU) or may be implemented by a process running on one of the APUs of the accelerator.In one aspect, each manager 420.m can operate on its own accelerator level subgraph, similar to the way that manager 220 of FIG. 2 can operate on graph 260 of FIG.

[0030] Thus, as described above, each node of graph 450 is part of a subgraph that is an APU level subgraph in the accelerator level subgraph. Graph 450 has a structure with connected nodes. Each node may have a data or object entity associated with it, such as a kernel, an assigned APU, a mailbox, or a timer, as described with reference to FIG. 3. Thus, each node in graph 450 is associated with a kernel that may be executed to process data stored in shared memory 440, for example. Each edge in graph 450 connects two nodes and represents a dependency between the kernels associated with the two nodes, as described with reference to FIG. 3. Furthermore, each node in graph 450 may have one or more mailboxes associated with it (e.g., stored in memory 440). A mailbox in a first node associated with a first kernel may be used by a second kernel (on which the first kernel depends) to leave a completion message indicating that the second kernel has completed, so that the first kernel can start its own execution as long as it is not yet waiting for the other kernels to complete their execution.

[0031] 3, subgraph 330 may be an APU level subgraph within the accelerator level subgraph of accelerator 410.1, and subgraph 340 may be an APU level subgraph within the accelerator level subgraph of accelerator 410.2. In this case, manager 420.1 may be configured to manage the execution of kernels B, F, G according to its own accelerator level subgraph, and manager 420.2 may be configured to manage the operation of kernels A, C, D, E, H according to its own accelerator level subgraph. In one aspect, node 310.7 has two mailboxes associated with it, one registered with node 310.2 and the other with node 310.3, where kernel G is waiting. When kernel B (of node 310.2) completes its execution, it leaves a completion message in the mailbox of node 310.7, where it is registered. Similarly, when kernel C (at node 310.3) completes its execution, it leaves a completion message in the mailbox of node 310.7 with which it is registered. Once these two completion messages are received, kernel G (at node 310.7) can immediately begin its own execution. Independently, managers 420.1 and 420.2 can update their respective portions of the graph (respective accelerator level subgraphs) to reflect the current operational state.

[0032] As mentioned before, graph 450 can dynamically change its structure as dependencies change during application runtime. For example, during the execution of kernel C, a kernel may realize or be notified that kernel G no longer depends on it. In such a case, kernel C can send a message to that effect to the mailbox registered with it at node 310.7. Kernel G, which is currently waiting for two mailboxes (one registered with kernel B and the other with kernel C), can then start its own execution once kernel B completes its execution. Meanwhile, during the execution of kernel C, a kernel may realize or be notified that kernel F should become dependent on it. In such a case, kernel C can generate a new mailbox in node 310.6 that is registered with it. Kernel F will then wait for that new mailbox in addition to the mailbox registered with kernel B, and only when it receives completion messages from both kernels B and C will kernel F start its own execution. Independently, the managers can update their respective parts of the graph to reflect the change in dependencies. For example, manager 420.1 monitoring subgraph 330 may add an edge between node 310.3 and node 310.6 to reflect that kernel F depends on kernel C, and manager 420.2 monitoring subgraph 340 may remove an edge between node 310.3 and node 310.7 to reflect that kernel G no longer depends on kernel C.

[0033] Managers 420.1-420.M may perform load balancing independently with respect to the portion of the graph they manage, or together with respect to all graphs 450. This may be done by monitoring timers associated with the nodes of graph 450 that are configured to measure the execution time of their respective kernels. Thus, at the end of each execution iteration of graph 450, based on analysis of the timer readings, each manager may repartition its own accelerator-level subgraph to load balance its operations. For example, referring to FIG. 3, assuming graph 300 is managed by one manager 420.1, kernels associated with subgraph 330 are assigned to APU-1 430.1.1 and kernels associated with subgraph 340 are assigned to APU-2 430.1.2. At the end of an iteration, based on analysis of the timer readings, manager 420.1 may find that APU-2 took more time to execute the kernel associated with its assigned subgraph 340 compared to the time APU-1 took to execute the kernel associated with its assigned subgraph 330. In such a case, manager 420.1 may decide to load balance the graph by repartitioning the graph. For example, repartitioning may result in node 310.5 becoming part of subgraph 330, and thus the kernel associated with node 310.5 will be executed by APU-1 230 in the next execution iteration of the graph.

[0034] In one aspect, load balancing operations may be performed collaboratively between the managers (possibly after independent load balancing has been performed as described above). For example, at the end of each iteration, each manager may compute an availability measure for each APU in its control (within its own portion of the graph) and may expose these APU availability measures to the other managers (e.g., by storing the availability measures in shared memory 440). The availability measure may be the time that the APU was idle during the iteration. For example, at the end of an iteration, managers 420.1 and 420.2 may compute availability measures for APUs 430.1.1-430.1.N and 430.2.1-430.2.N, respectively. If the availability metric of APU 430.1.1 is sufficiently high and the availability metric of APUs 430.2.1-430.2.N is sufficiently low, manager 420.1 may lock graph 450 and then repartition the graph to change the allocation of one or more kernels running on APUs 430.2.1-430.2.N to the available APU 430.1.1. Manager 420.2 may indicate whether manager 420.1 has permission to perform the repartitioning. Other mechanisms may be used to cooperatively load balance graph 450. In one aspect, the manager's calculation of the availability metric of the APUs in its own portion of the graph may be done after (and taking into account) the independent load balancing of its own portion of the graph.

[0035] 5 is a block diagram of an example system 500 illustrating distributed management of application runtime across network boundaries, upon which one or more features of the present disclosure may be implemented. System 500 may include multiple accelerators 510.1-510.M similar in operation to multiple accelerators 410.1-410.M of system 400 of FIG. 4. However, in system 500, accelerators 510.1-510.M are remote from one another and each is connected to a respective memory 510.1-510.M and a respective processor 560.1-560.M local thereto. As shown, accelerators 510.1-510.M, memories 510.1-510.M, and processors 560.1-560.M are communicatively coupled via network 570. Any processor, e.g., 560.1, can utilize an API exposed by any manager, e.g., 520.1, to initiate a graph that defines the processing flow of an application executed by system 500. The initiated graph may then be stored in memory 540.1 accessible by accelerator 510, and copies of the graph can be sent over network 570 and stored in memories 560-510.2.M accessible by accelerators 510.1-560.2.M.

[0036] In one embodiment, the operation of system 500 is similar to that of system 400. However, system 500 must maintain a consistent representation of the graph through its copies 550.1-550.M. For example, at the end of each iteration, the copies of graphs 550.1-550.M may be updated by their respective managers 520.1-520.M (e.g., each manager may update the part of the graph it owns), and then the copies may be reconciled. In one embodiment, one processor, e.g., 560.1, may be responsible for reconciling the graphs. To that end, processors 560.2-560.M may be configured to send their copies of the graph to processor 560.1, which reconciles copies 550.1-550.M and returns one reconciled graph that is stored in the respective memory 540.1-540.M. In system 500, such communications by APUs that are remote from one another (e.g., when a kernel running on one APU sends a message to a mailbox associated with a kernel running on another APU) may contribute latency, however such latency is limited by the bandwidth of network 570 and is distinct from the latency contributed by CPU-APU communications.

[0037] FIG. 6 is a flow chart of an example method 600 for accelerator resident runtime management, based on which one or more features of the present disclosure may be implemented. Method 600 may be performed by an accelerator resident manager, such as manager 220 of FIG. 2, or any of managers 420.1-420.M of FIG. 4 or managers 520.1-520.M of FIG. 5. Method 600 may receive a representation of the processing flow of an application, including kernels and their respective dependencies, at step 610. As discussed above, the representation may be represented by a graph, such as graph 300 of FIG. 3. At step 620, method 600 may assign kernels to APUs managed by the manager. Then, at step 630, method 600 may launch kernels on their assigned APUs for execution in an iteration according to the kernels' respective dependencies. If during an iteration there is a change in the dependencies (either the termination of a dependency or the creation of a new dependency), method 600 may notify the affected kernels at step 640. For example, if a dependency of a first kernel on a second kernel is terminated during an iteration, a message indicating the terminated dependency can be sent by the second kernel to the first kernel. Upon receiving that message, the first kernel can immediately begin execution (as long as it is not yet waiting for another kernel to complete). In another example, if a new dependency of a first kernel on a second kernel is created during an iteration, a mailbox can be created by the second kernel that is waited for by the first kernel. When the second kernel completes its own execution, it sends a completion message to that mailbox. At the end of an iteration, in step 650, method 600 can update the representation. For example, if a change in a dependency occurs during an iteration (either the end of a dependency or the creation of a new dependency), method 600 can change the allocation of kernels to APUs based on the change.Thus, in the graph representation 300, the method 600 can repartition the graph, changing the structure of the subgraphs 330, 340, and effectively changing the allocation of some of the kernels A-H 310 to either APU-1 230 or APU-2 240.

[0038] It should be understood that many variations are possible based on the disclosure herein, and although features and elements have been described above in particular combinations, each feature or element can be used alone without the other features and elements, or in various combinations with or without the other features and elements.

[0039] The provided methods can be implemented in a general purpose computer, processor, or processor core. Suitable processors include, by way of example, general purpose processors, special purpose processors, conventional processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGA) circuits, any other type of integrated circuit (IC), and / or state machines. Such processors can be manufactured by configuring a manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data including netlists (instructions that can be stored on a computer readable medium). The results of such processing can be a mask work, which is then used in a semiconductor manufacturing process to manufacture a processor implementing aspects of the embodiments.

[0040] The methods or flow charts provided herein may be implemented in a computer program, software, or firmware embodied in a non-transitory computer-readable storage medium for execution by a general purpose computer or processor. Examples of non-transitory computer-readable storage media include read only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

Claims

1. 1. A method of runtime management by an accelerator resident manager, comprising: receiving, by the manager, a representation of a process flow of an application including a number of kernels and their respective dependencies; the manager assigning the plurality of kernels to one or more APUs of an accelerator managed by the manager; the manager launching the kernels on their assigned APUs and executing them iteratively according to their respective dependencies. method.

2. if a dependency of a first kernel on a second kernel terminates during the iteration, the second kernel sending a message to the first kernel, the message indicating the terminated dependency.

2. The method of claim 1.

3. and at the end of the iteration, the manager reallocating the kernels to the APUs based on completed dependencies. The method of claim 2.

4. generating, by the second kernel, a mailbox used by the first kernel to receive a completion message from the second kernel if a dependency of the first kernel on the second kernel is generated during the iteration.

2. The method of claim 1.

5. at the end of the iterations, the manager reallocating the kernels to the APUs based on the generated dependencies. The method of claim 4.

6. the manager load-balancing execution of the kernels by reallocating a subset of the kernels to different APUs based on execution times of the kernels on the assigned APUs during the iteration.

2. The method of claim 1.

7. the allocating includes allocating a portion of the plurality of kernels to a second accelerator resident manager, the second manager allocating a portion of the plurality of kernels to one or more APUs of a second accelerator managed by the second manager; the launching includes the second manager launching some of the kernels on their assigned APUs and iteratively executing them according to their respective dependencies.

2. The method of claim 1.

8. the manager and the second manager are remote from each other; the invoking by the manager and the invoking by the second manager are performed according to dependencies contained in respective copies of the representation, including verifying copies of the representation. The method of claim 7.

9. the representation is a graph; Each node of the graph is associated with one of the kernels and one of the APUs; Each edge of the graph represents one of the respective dependencies.

2. The method of claim 1.

10. each node of the graph is associated with a timer that measures the execution time of a kernel associated with the node; 10. The method of claim 9.

11. 1. A system for runtime management by an accelerator resident manager, comprising: At least one processor; A memory for storing instructions, The instructions, when executed by the at least one processor, receiving, by the manager, a representation of a process flow of an application including a number of kernels and their respective dependencies; the manager assigning the plurality of kernels to one or more APUs of an accelerator managed by the manager; the manager launching the kernels on their assigned APUs and executing them iteratively according to their respective dependencies; causing the system to system.

12. The instruction: if a dependency of a first kernel on a second kernel terminates during said iteration, said second kernel sending a message to said first kernel, said message indicating the terminated dependency; At the end of the iterations, the manager reallocates the kernels to the APUs based on completed dependencies; causing the system to The system of claim 11.

13. The instruction: generating, by the second kernel, a mailbox used by the first kernel to receive a completion message from the second kernel if a dependency of the first kernel on the second kernel is generated during the iteration; At the end of the iterations, the manager reallocates the kernels to the APUs based on the generated dependencies; The system of claim 11 , further comprising:

14. The instruction: the manager causes the system to load balance execution of the kernels by reassigning a subset of the kernels to different APUs based on execution times of the kernels on the assigned APUs during the iteration. The system of claim 11.

15. the allocating includes allocating a portion of the plurality of kernels to a second accelerator resident manager, the second manager allocating a portion of the plurality of kernels to one or more APUs of a second accelerator managed by the second manager; the launching includes the second manager launching some of the kernels on their assigned APUs and iteratively executing them according to their respective dependencies. The system of claim 11.

16. the manager and the second manager are remote from each other; the invoking by the manager and the invoking by the second manager are performed according to dependencies contained in respective copies of the representation, including verifying copies of the representation.

16. The system of claim 15.

17. 1. A computer-readable storage medium comprising instructions executable by at least one processor to perform a method for accelerator resident manager runtime management, the method comprising: The method comprises: receiving, by the manager, a representation of a process flow of an application including a number of kernels and their respective dependencies; the manager assigning the plurality of kernels to one or more APUs managed by the manager; the manager launching the kernels on their assigned APUs and executing them iteratively according to their respective dependencies. A computer-readable storage medium.

18. if a dependency of a first kernel on a second kernel terminates during said iteration, said second kernel sending a message to said first kernel, said message indicating the terminated dependency; and at the end of the iteration, the manager reallocating the kernels to the APUs based on completed dependencies.

20. The computer readable storage medium of claim 17.

19. generating, by the second kernel, a mailbox used by the first kernel to receive a completion message from the second kernel if a dependency of the first kernel on the second kernel is generated during the iteration; and at the end of the iterations, the manager reallocates the kernels to the APUs based on the generated dependencies.

20. The computer readable storage medium of claim 17.

20. the manager load-balancing execution of the kernels by reallocating a subset of the kernels to different APUs based on execution times of the kernels on the assigned APUs during the iteration.

20. The computer readable storage medium of claim 17.

Citation Information

Patent Citations

  • Managing Data Flow in Heterogeneous Computing

    US20180074727A1

  • Technologies for offloading acceleration task scheduling operations to accelerator sleds

    US20180150298A1