Dynamic Communication Architecture for Distributed Heterogeneous Accelerators

Abstract

A dynamic communication architecture for distributed heterogeneous accelerators is disclosed. A system and method for a dynamic communication architecture are provided, which can include distributed heterogeneous accelerators and operations for mapping the corresponding distributed virtualized accelerators. The mapping of the corresponding distributed virtualized accelerators is performed on, for example, a memory-mapped multi-accelerator communication architecture to achieve efficient and flexible data sharing. By analyzing workflow communication, the mapping can be dynamically adjusted according to the running workload.

Description

Background Technology

[0001] With the convergence of artificial intelligence (AI), high-performance computing (HPC), and data analytics (DA) workflows, multiple distinct workflows need to efficiently execute and / or concurrently complete a task that potentially runs on multiple heterogeneous accelerators, central processing units (CPUs), and networks. Centralized CPU accelerator memory mapping schemes have been used to rigidly and statically map accelerators to communication architectures. However, the scaling of disparate, large-scale workflows (especially in AI / HPC / DA workflows, for example) often utilizes multiple heterogeneous accelerators for efficient communication. However, communication using multiple heterogeneous accelerators is inconsistent and often slow, which typically leads to performance degradation and overall system inefficiency. Attached Figure Description

[0002] This disclosure is described in detail with reference to the following figures, based on one or more various examples of the disclosed technology. These figures are for illustrative purposes only and depict only typical or exemplary aspects of the disclosed technology.

[0003] Figure 1A-1B The diagram illustrates a communication architecture system that depicts the execution location of a workflow and the execution of workflow commands, based on the example described herein.

[0004] Figure 2 The illustration depicts a memory-mapped communication architecture that illustrates memory mapping between accelerators and corresponding communication between accelerators, according to one or more examples described herein.

[0005] Figure 3 The illustration shows a general abstraction for programmers that exposes a unified interface to simplify programming tasks, based on one or more examples described in this article.

[0006] Figure 4 The illustration shows a mapping of virtualization accelerators on top of a virtualization communication architecture, including data exchange between virtualization accelerators, according to one or more examples described herein.

[0007] Figure 5 The illustration shows an implementation of an underlying memory mapping across multiple nodes according to one or more examples described herein.

[0008] Figure 6 This is a flowchart illustrating an algorithm for mapping a self-adjusting virtual accelerator to improve the efficiency of a communication architecture system (e.g., by reducing data movement), based on one or more examples described herein.

[0009] Figure 7 It is a computational component that can be used to implement various features of the examples described in this disclosure.

[0010] Figure 8 A block diagram depicts a computer system in which various features of the examples described herein can be implemented.

[0011] These accompanying drawings are not exhaustive and do not limit this disclosure to the precise form disclosed. Detailed Implementation

[0012] It should be understood that the accompanying drawings and specifications of the disclosed technology may have been simplified to schematically illustrate elements relevant to a clear understanding of the disclosed technology, and other elements found in typical accelerator systems, typical communication architecture systems, typical methods using accelerator systems, or typical methods using communication architecture systems have been omitted for clarity. Those skilled in the art will recognize that other elements may be needed and / or required to implement the disclosed technology. However, since these elements are well-known in the art and because they do not contribute to a better understanding of the disclosed technology, a discussion of these elements is not provided herein. It should also be understood that the accompanying drawings included herein provide only a schematic representation of the structure of the disclosed technology, and structures within the scope of the disclosed technology may include structures different from those shown in the drawings. Reference will now be made to the accompanying drawings, in which similar structures are provided with similar reference numerals.

[0013] Before explaining at least one example of the disclosed technology in detail, it should be understood that the inventive concepts described herein are not limited to their application to the construction details or component arrangements set forth in the following specification or shown in the accompanying drawings. It should also be understood that the wording and terminology used herein are for descriptive purposes only and should not be considered restrictive.

[0014] It should also be understood that any of the features described can be used alone or in combination with other features. Other inventive devices, structures, apparatuses, systems, methods, features, and advantages will be or become apparent to those skilled in the art upon examination of the accompanying drawings and detailed description herein. All such additional devices, structures, apparatuses, systems, methods, features, and advantages are intended to be protected by the appended claims.

[0015] For the purposes of this disclosure, the phrase "communication architecture" refers to a high-level definition that hides the complexity of the backend, and the exact implementation may depend on the hardware.

[0016] For the purposes of this disclosure, the phrase “generic abstraction” refers to a generic interface in which programmers can use the same (generic) interface to coordinate communication, regardless of the topology between accelerators.

[0017] It should be noted that the terms “optimized,” “optimal,” etc., used herein can be used to indicate achieving or attaining the most efficient or perfect performance possible. However, as those skilled in the art who read this document will recognize, perfection is not always achievable. Therefore, these terms can also include achieving or attaining the best or most practically feasible performance possible under given conditions, or achieving or attaining better performance than could be achieved using other settings or parameters.

[0018] As described in more detail below, the disclosed examples of the technology map virtualized accelerators on top of a memory-mapped multi-accelerator communication architecture and dynamically adjust the mapping for the running workload in terms of latency (execution time) and throughput. Therefore, the disclosed examples of the technology select the optimal communication mechanism between system components. In doing so, the system considers latency, bandwidth / throughput, and optional data transfer modes when selecting the communication architecture (e.g., between system components such as the CPU and multiple accelerators), while coordinating the communication architecture with the underlying processing mechanism (i.e., the workflow code execution component). During runtime, adjustments are made to workflow changes to simplify the management of these system components.

[0019] For the purposes of this disclosure, an accelerator is a computing machine that includes a coprocessor connected to a central processing unit (CPU) capable of accelerating the execution of specific computationally intensive kernels, thereby accelerating overall execution according to Amdahl's Law. End applications contain multiple computing kernels, and the properties of these parts are better executed by a specific accelerator, which can also be other computing machines, such as Field Programmable Gate Arrays (FPGAs), Coarse-grained Reconfigurable Architectures (CGRAs), Graphics Processing Units (GPUs), Neural Processing Units (NPUs), Data Processing Units (DPUs), Visual Processing Units (VPUs), Application-Specific Integrated Circuits (ASICs), Tensor Processing Units (TPUs), SmartNICs (Smart Network Interface Cards), or even CPUs. Kernels are code that runs inside an accelerator. Therefore, kernels typically run uniquely on each of these accelerators. While kernels can be replicated, they can be partitioned across each accelerator. However, they are typically uniquely mapped to different, specific accelerators.

[0020] The communication architecture described in any example of the disclosed technology can be applied between heterogeneous accelerators of various types of computing machines, such as CPUs, FPGAs, GPUs, etc., as described above. The communication architecture can be applied between computing machines of similar categories (e.g., GPU-GPU) or between different combinations of computing machines of different categories (e.g., GPU-FPGA). Alternatively, the communication architecture described in any example of the disclosed technology can be applied between homogeneous accelerators.

[0021] As mentioned above, communication using multiple heterogeneous accelerators can be inconsistent and often slow, typically leading to performance degradation and overall system inefficiency. These drawbacks are even more prevalent in large-scale computing systems, including those for artificial intelligence (AI), high-performance computing (HPC), and data analytics (DA) workflows. To overcome these drawbacks, this paper presents a solution using a dynamic communication architecture to simplify system programming and management.

[0022] Large-scale computing systems utilize a collection of heterogeneous computing machines / accelerators (e.g., FPGAs, GPUs, etc.) with varying performance requirements, workload classifications, and capabilities (e.g., memory capacity, functionality, etc.) to execute a hybrid collection of heterogeneous workloads within a workflow (e.g., stateless function calls, long-running services, batch data analysis, interactive development and testing, etc.). Workloads execute instructions through code execution commands controlled by the workflow. Workloads within a workflow can be enabled for each instruction, for example, by allocating workloads to different computing resources within the computing system.

[0023] Different workflow tasks exist, suited to run on different heterogeneous accelerators due to their varying performance requirements and workload classifications. For example, GPUs are suitable for throughput-oriented workloads, while FPGAs are suitable for latency-sensitive workloads. Due to the mix of these workloads and system efficiency, overall performance in heterogeneous accelerator environments should be improved. As workflows extend to different accelerators, even across nodes, multiple accelerators need to communicate effectively. Therefore, the examples of the disclosed techniques address how to abstract the low-level, detailed communication mechanisms involved in these different workflow tasks into a unified communication interface, thereby reducing programming effort and without sacrificing (the overall system efficiency otherwise gained through the use of heterogeneous accelerators).

[0024] Using heterogeneous accelerators with different communication methods can lead to compatibility issues, making efficient communication for each component cumbersome. Therefore, a single interface is needed to provide uniformity, freeing users from worrying about the underlying communication details. This ideal, universal abstraction provides programmers with a high level of visualization, simplifying the programming process. With this universal abstraction, many elements are virtualized, such as computing power, PCIe, networking, and kernel execution priorities. These unified abstractions can also encompass internal communication occurring between computing machines.

[0025] Analysis of kernel and workflow communication can also enable self-regulating and dynamically reconfigurable communication architectures. Specifically, accelerators will feature affinity detection, binding, and unbinding of kernel and data to reduce data movement within the cluster. This approach also allows for variations in the protocols or protocol semantics of data exchange. Through this self-management and scalability, servers can achieve greater robustness and efficiency.

[0026] Large-scale computing systems, such as those incorporating AI / HPC / DA workflows, can include a large number of nodes (e.g., tens of thousands), each with associated resources. Some large computing systems may utilize processing resources, memory, accelerators, and other resources that can be allocated to users at levels requested by the user. Operating systems and kernels can run on some or all of the nodes, and in some cases, different types, versions, or instances of kernels may run on different nodes within the system. System nodes (i.e., within a node) and their resources can communicate with each other through distributed inter-node communication on the fabric.

[0027] As discussed in more detail below, the unification of inter-node and intra-node, memory-mapped, decentralized accelerator communication mechanisms results in a new dynamic communication architecture. Inter-node communication refers to communication between accelerators across different servers, while intra-node communication refers to communication between accelerators within a specific server. This unified and dynamic memory-mapped multi-accelerator communication architecture provides programmers with a common abstraction for compute accelerators both within and across nodes (e.g., heterogeneous). In other words, the dynamic communication architecture allows programmers to use the same / unified interface to coordinate communication regardless of the various topologies between accelerators and without needing to know the communication details within the communication architecture under the virtualized accelerator. This underlying communication architecture and its unified interface provide a high-level definition that can be implemented in various ways under the virtualized accelerator and optimized for data locality and data flow awareness, as discussed in detail below. Since all communication details can remain hidden from and unexposed by the programmer, instead, this communication can be conveyed by the programmer through the set of APIs used by that programmer.

[0028] Regarding memory-mapped communication, each accelerator has its own memory, and one accelerator can read from or write to the memory of another accelerator. Each accelerator's memory is exposed and mapped, allowing accelerators to write to (or read from) the memory of another accelerator. The accelerator communication mechanism is inter-node, and therefore there is no central entity. Due to this decentralization, one server communicates with another without going through a central entity. This contrasts with a coordinating central server that receives requests from one server and routes those requests to another. This centralized approach typically involves a host entity to coordinate every communication between different accelerators working across different servers. For example, an accelerator wanting to communicate with another accelerator might go through a host. This additional coordination using a host can be very cumbersome and restrictive, as it also lacks direct point-to-point communication. Furthermore, the mapping in a centralized approach is static during execution, thus requiring greater flexibility in the communication architecture to effectively utilize the diverse capabilities of the underlying computing hardware.

[0029] Another advantage of using a communication architecture that includes virtual accelerators is that each accelerator, or a portion thereof, can function as or appear as multiple accelerators. This allows for flexibility when sharing small portions of an accelerator with other accelerators. Using virtualized accelerators in the communication architecture also allows for dynamic changes in mapping (e.g., the size of a particular virtual accelerator, the amount of compute power a virtual accelerator has depending on data locality, and the data communication patterns used). This virtualization configuration also allows for management capabilities to visualize what types of data will be processed, how many accelerators are needed, how data will be routed to virtual accelerators, etc. Computational elements on top of accelerators and other computing machines can all be virtualized. Mapping virtualized accelerators on top of the communication network simplifies management. The accelerator itself is virtualized for multi-tenancy purposes (for both virtualized accelerators and physical / bare-metal accelerators), and this virtualized accelerator is mapped to the underlying virtualized communication architecture. Once the virtual accelerator is mapped to the underlying communication architecture, programmers can transparently build virtual clusters on top of various communication mechanisms. The communication network can also take into account data flow and data movement, as well as the current characteristics of the workload. For data retention, current characteristics can be associated with read-only, modify, delete later, and read again. Virtualization can enhance security by using data isolation and a zero-trust data access model across clients, especially in multi-tenant execution scenarios.

[0030] Figure 1A and 1B Example communication architecture systems 100a and 100b are illustrated respectively, which depict the execution location of workflow (i.e., program code) 110 and the execution of workflow commands. Figure 1A and 1B The details of how workflow 110 is divided into different blocks (i.e., workflow code execution units 112a-112f) are shown, which are mapped to CPU 122a and different accelerators 122b-122f. Figure 1A and 1B The dashed lines shown indicate the execution locations of the workflow. In other words, dashed lines represent logical operations performed on CPU 122a and accelerators 122b-122f. Conversely, solid lines represent the execution of workflow commands (from the program or processing / computing unit). In other words, solid lines represent the actual communication occurring between the computing machines. Figure 1A In the communication architecture system 100a shown, all accelerators 122b-122f are directly connected to CPU 122a. This architecture indicates that some communication to accelerators 122b-122f must first pass through CPU 122a (i.e., through workflow code execution component 112a). Alternatively, Figure 1AEach of the workflow code execution units 112a-112f in the illustrated workflow 110 can first pass through CPU 122a and then to the corresponding accelerator (any one or more of accelerators 122b-122f). Using this alternative routine, workflow code execution units 112a-112f can communicate with CPU 122a, and then CPU 122a communicates with accelerators 122b-122f. Therefore, workflow code execution units 112a-112f communicate with accelerators 122b-122f via CPU 122a. Figure 1B Different communication architecture systems 100b are shown, in which accelerators 122b-122f are interconnected (and communicate) with each other in the form of a typical serial chain (i.e., except for accelerators 122d and 122e, which are connected in parallel between accelerators 122c and 122f). It should be noted that, although... Figure 1A and Figure 1B The mapped workflow is shown. A workflow manager (not shown) can be used to coordinate workflow deployments on different computing machines.

[0031] Now refer to Figure 2 , Figure 2 The illustration shows an example memory-mapped communication architecture 200. The memory-mapped communication architecture 200 includes memory mapping between CPU memory 232a and accelerator memories 232b-232f, and corresponding communication between CPU 122a and accelerators 122b-122f. CPU 122a and accelerators 122b-122f can be... Figure 1B The communication architecture system 100b shown therein. Figure 2 In the diagram, solid lines represent actual communication between accelerators, and dashed lines represent memory-mapped communication between accelerators. In communication architecture 200, accelerator memory is exposed to the memory of other accelerators for communication between them, allowing read, write, and modification of memory from another computing machine. As an example, accelerator memory 232b from accelerator 122b is reading a block from CPU memory 232a from CPU 122a. The block from CPU memory 232a is being copied or transferred to accelerator memory 232b. And the block from accelerator memory 232b is being copied or transferred to accelerator memory 232c from accelerator 122c.

[0032] in spite of Figure 2 The example memory mapping communication architecture 200 shown illustrates how a specific memory mapping is established on a workflow-by-workflow basis, as an example of the disclosed techniques. Figure 1A and Figure 1B The workflow in the document establishes a workflow from the CPU to another accelerator.

[0033] This establishes a communication architecture across various processing elements, including CPUs and different accelerators, to enable workflows to be executed efficiently. Mapping workflows and accelerators presents numerous challenges, particularly for workflows representing, for example, large AI models that can be mapped onto CPU and accelerator memory. Therefore, the ability to communicate efficiently across these computing machines using shared memory is highly advantageous. The disclosed techniques illustrate mapping accelerator memory to establish this efficient communication architecture for workflow execution.

[0034] As briefly discussed above, the examples of the disclosed technologies also point to a unified programming interface for programmers to simplify their programming work. Users do not need to know the communication details under the unified interface. The unified interface and its underlying communication architecture provide a high-level definition that can be implemented in various ways at the lower level and optimized for data locality, data, and workflow awareness. In the example, within a node, there is a memory mapping for shared virtual memory, and specifically for shared memory for, for example, FPGAs and GPUs, which are two types of accelerators that can share a single virtual address. This allows FPGAs and GPUs to transparently access each other's memory and maintain consistency. Alternative underlying communication architectures can include point-to-point communication via a PCIe bus / switch. In the example of a point-to-point setup, one accelerator can initiate a data request or push data to other accelerators without host intervention.

[0035] This memory mapping technique can also be used between nodes. For example, the underlying communication architecture can use a memory mapping mechanism across nodes. This can be implemented using Remote Direct Memory Access (RDMA), or, if the architecture supports such an interface, it can be achieved by using load and store semantics for better efficiency and ease of programming. An alternative implementation can use point-to-point messaging that is passed directly between two nodes for better scalability and reliability.

[0036] This implementation, using an underlying communication architecture, can address data flow patterns (e.g., determining the next device to access data). The underlying communication architecture achieves this by analyzing the workflow and ensuring that the data resides in memory accessible to the accelerator that will process it next.

[0037] Regarding providing a general abstraction, regardless of the underlying mechanisms, a topology-aware scheme (at the interface) allows programmers to automatically determine the optimal communication method. For example, programmers can easily choose between point-to-point (distributed) communication, direct communication, or centralized communication based on the data exchange pattern. In the example, if the data exchange is simple and occurs between two accelerators, point-to-point communication is chosen. If the data exchange is fine-grained and requires coordination across multiple accelerators, centralized communication is chosen. Therefore, it is clear that different communication mechanisms are suitable for different workflows, all without requiring host intervention at the programmer's interface. This is why a general abstraction is needed—to simplify programming.

[0038] Regarding the virtualization of communication architecture, when multiple users are running on a single physical hardware, virtualization provides isolation for each user and can also provide different scheduling policies for, for example, higher-priority jobs. Furthermore, this can be extended to different compute nodes. For example, fabric-attached memory and RDMA capabilities with available fabric allow for low-latency communication between different compute nodes.

[0039] Now refer to Figure 3 , Figure 3 The diagram illustrates a sample general abstraction for programmers 300, used to expose a unified interface that simplifies programming tasks. The diagram shows kernel 1 360 (or workflow) and data exchanged to kernel 2 370 (via data exchange 361, and ultimately via data exchange 366 provided by the unified interface 365) for direct use by the programmer. The programmer cannot see what lies between kernel 1 360 and kernel 2 370. This unseen internal workings are provided for synchronization within a node or with another node (via inter-node queries and subsequent communication). Figure 3As shown, the decision 362 determines whether data exchange is intra-node or inter-node. If the data exchange is intra-node 363, the data is communicated using shared virtual memory or peer-to-peer transmission. If the data exchange is inter-node 364, the data is communicated using message passing, shared memory, or partitioned global address space. The monitoring unit 368 monitors the exchange characteristics 367 (e.g., data locality) within the data exchange 361. The monitoring unit 368 detects how much data must be moved and where it must be moved. Based on the exchange characteristics 367 of the monitored data exchange 361, the monitoring unit then sends its output as a decision 369 regarding whether data locality within the workflow needs improvement or adaptation to the new workflow. If data locality within the workflow does not need improvement or adaptation to the new workflow, the monitoring unit 368 continues to monitor the data exchange 361. If data locality within the workflow needs improvement or adaptation to the new workflow needs improvement, the matching degree of the communication mechanism or accelerator is adjusted.

[0040] To implement self-regulation, monitoring unit 368 can analyze the semantics of kernel calls (e.g., how much data it requires, when its kernel is invoked, etc.) and data communication patterns in real time. For example, it can identify from which accelerators or nodes the kernel pushes or pulls data, how much data is transferred, and / or the frequency of data transfer. Furthermore, monitoring unit 368 can analyze the data matching degree of accelerators with respect to memory pools. For example, it can determine whether data is transferred between accelerators or remains on a single accelerator. This determination made by monitoring unit 368 can dynamically bind or unbind virtual accelerators used for kernel execution to improve data locality in the workflow and retain data on accelerators for as long as possible, or adapt to new workflows.

[0041] To select the optimal communication method for intra-node (363) or inter-node (364) communication, a model (analytical or graph-based) can be used to choose the best communication mechanism. This model can consider latency, bandwidth, and / or data transmission patterns to select the optimal communication architecture. During runtime, the model can be updated to reflect the system state.

[0042] Referring to the virtualization implementation described above, virtualization accelerators can be mapped onto a communication architecture. This communication architecture can support multi-tenancy for both virtualization accelerators and bare-metal accelerators. An example of this virtualization can be used for GPU virtualization, which can be used for multi-instance GPUs, multi-process services, and other similar technologies. GPUs can also use container technology to achieve more flexible sharing at the software level. These software technologies enable flexible sharing. Another example of this virtualization can be for FPGA virtualization, where it can be implemented using multi-dynamic part reconfiguration for multi-tenancy. Alternatively, FPGA shells can be used to achieve more resource sharing. To accommodate some accelerators that may not be very supportive of virtualization or multi-tenant sharing (such as CGRA), the illusion of virtual accelerator properties can be created through static partitioning (i.e., resulting in multi-tenancy). In other words, statically partitioned accelerators create the illusion that the accelerator has virtual accelerator properties.

[0043] Once virtual accelerators are mapped to the underlying communication architecture, programmers can transparently build virtual clusters on various communication mechanisms. These accelerators may have different available communication hardware and capabilities. Accelerators (virtual and physical) are mapped to the communication architecture. Accelerators register their available communication hardware (PCIe, architectures such as NVLink and Infiniband) and communication capabilities (bandwidth / latency to the host and other accelerators) to help select the optimal communication method.

[0044] When a request occurs, different communication architectures are instantiated on demand, which reduces unnecessary waste of hardware resources. Each virtual accelerator registers its communication architecture upon instantiation. When a communication architecture is instantiated depends on the data processing mode and mechanism. Each virtual accelerator that registers a different communication architecture is mapped to the same or different physical hardware.

[0045] The data processing mechanism used by the kernel in the accelerator can be streaming and / or non-streaming, and any data retention depends on the workflow. This data retention can have the following characteristics: (1) read-only; (2) modify after reading; (3) delete after reading, etc. The communication architecture transparently handles data reading mechanisms and data retention, etc.

[0046] If users require higher security, then the use of virtualization will typically ensure security by enabling stricter sharing. For example, multi-instance GPU sharing enables greater isolation compared to containers running on GPUs. Enabling data isolation for virtualization accelerators sharing the same physical accelerator and enabling tenant isolation for large-scale workflows can occupy an entire node. In other words, the same physical node can occupy an entire compute node.

[0047] Now refer to Figure 4 , Figure 4 This diagram illustrates an example mapping of virtualization accelerators 422a-1, 422a-2, 422b-1, 422b-2, 422c-1, 422c-2, 422c-3, and 422d-1 on a virtualized communication architecture 400, including data exchange 490ac-1, 490ac-2, 490bc-1, 490bc-2, and 490bd-1 between the virtualization accelerators. In the example, data exchange 490ac-1 is provided between virtualization accelerators 422a-1 and 422c-1. The diagram also shows bare-metal accelerators 422a, 422b, 422c, and 422d, and the virtualization of these bare-metal accelerators. For example, by virtualizing accelerators 422a-1 and 422a-2 on bare-metal accelerator 422a, it appears that multiple accelerators exist in place of bare-metal accelerator 422. Different communication methods (e.g., shared memory on the host 444, shared memory on one of the accelerators 442, PCIe switch 448, and message passing (for cross-node) 446) can be used to allow the virtualization accelerator to communicate seamlessly with other virtualization accelerators or bare metal accelerators. Therefore, even when using a virtualization accelerator, it will still use the underlying communication architecture for communication.

[0048] Now refer to Figure 5 , Figure 5An example of an underlying memory mapping implementation 500 across multiple different nodes (i.e., node No.1 551 and node No.2 555) is illustrated. As shown, different virtualization regions (i.e., virtualization accelerators 550) exist within the physical accelerator. For example, in FPGA 553, there are Dynamic Partial Reconstruction (DPR) regions (i.e., DPR No.1 and DPR No.2). Communication can occur within a node between two accelerators (i.e., GPU 552 and FPGA 553), and more specifically between the Multi-Instance GPU (MIG) (within GPU 552) and DPR No.1 (within FPGA 553). This communication can be accomplished via shared virtual memory 542. Communication across different nodes between the Multi-Process Service (MPS) within GPU 552 (within node No.1 551) and the MPS within GPU 556 (within node No.2 555) can be accomplished via shared memory 544. Furthermore, communication across different nodes between FPGA 553 (within node No.1 551) and CGRA 557 (within node No.2 555) can be achieved via message passing communication 546. Various communications sharing virtual memory 542, shared memory 544, and message passing communication 546 are considered to be on the underlying network 540. Given that the virtualization accelerators are on the underlying network, programmers do not need to know exactly what is happening underneath. Therefore, they can focus on, for example, which two types of accelerators are needed to communicate with each other.

[0049] Figure 6 This is a flowchart illustrating algorithm 600 for self-adjusting virtual accelerator mapping to improve the efficiency of a communication architecture system (e.g., by reducing data movement). The diagram shows how an example of data pattern analysis and optimization is performed on the overall system configuration to improve the efficiency of the communication architecture system. Based on the communication performance characteristics (e.g., latency, bandwidth, shared state, etc.) between different virtual accelerators, frequent data exchange can be checked between several virtual accelerators (step 602). If the answer is no, the model is updated to reflect the system characteristics (e.g., latency, bandwidth, shared state, etc.) (step 606); if the answer is yes, it is determined whether the virtual accelerators (or virtual clusters) are physically close to each other (step 604). If the answer is no, then the model is (possibly) updated again to reflect the system characteristics (e.g., latency, bandwidth, shared state, etc.) (step 606); if the answer is yes, the virtual accelerators in the virtual cluster dynamically move closer to each other to improve data locality (step 608).

[0050] Examples of the techniques disclosed herein also relate to systems and methods for providing dynamic communication architectures that can be used with multiple different workflows on a large scale. These multiple different workflows can include, for example, artificial intelligence (AI), high-performance computing (HPC), and data analytics (DA) workflows. Dynamic communication architectures can also be used with multiple heterogeneous accelerators in a decentralized (i.e., serverless) configuration. Heterogeneous accelerators have different capabilities, such as in terms of memory capacity or support for different data streams. The different accelerators' support for different data streams can relate to their ability to handle different workloads suitable for different purposes, performance, and energy efficiency factors.

[0051] Various examples of the disclosed techniques employ the ability to map distributed heterogeneous accelerators using corresponding distributed virtualization accelerators. The mapping of the corresponding distributed virtualization accelerators is performed on a multi-accelerator communication architecture, such as memory mapping, to achieve efficient and flexible data sharing. To this end, kernels are deployed on different distributed accelerators, and communication methods / mechanisms are configured according to the accelerator communication characteristics to achieve optimal system efficiency. By analyzing workflow communication, the mapping can be dynamically adjusted for the running workload. More specifically, the mapping can be dynamically changed based on data communication patterns (e.g., using communication performance characteristics such as latency and / or throughput) and / or data locality.

[0052] Figure 7 The illustration shows an example computing component that can be used to implement a communication architecture system according to various aspects of this disclosure. Reference is now made to... Figure 7 The computing component 700 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 7 In an example implementation, computing component 700 includes hardware processor 702 and machine-readable storage medium 704.

[0053] The hardware processor 702 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in the machine-readable storage medium 704. The hardware processor 702 may fetch, decode, and execute instructions, such as instructions 706-710, to control processes or operations for implementing a dynamically modular and customizable computing system. As an alternative to or supplement to retrieving and executing instructions, the hardware processor 702 may include one or more electronic circuits comprising electronic components, such as FPGAs, ASICs, or other electronic circuits, for the function of executing one or more instructions.

[0054] A machine-readable storage medium, such as machine-readable storage medium 704, can be any electronic, magnetic, optical, or other physical storage device that contains or stores executable instructions. Therefore, machine-readable storage medium 704 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In this example, machine-readable storage medium 704 can be a non-transitory storage medium, where the term "non-transitory" excludes transient propagation signals. As described in detail below, machine-readable storage medium 704 can be encoded with executable instructions such as instructions 706-710.

[0055] Hardware processor 702 can execute instructions 706 to map multiple workflow code execution units to multiple system units. Workflow code execution units are different segments / blocks extracted from a workflow. These different blocks are mapped to system units. System units can be, for example, CPUs, accelerators, GPUs, etc.

[0056] Hardware processor 702 may execute instructions 708 to determine a communication performance characteristic level associated with each of a plurality of workflow code execution units. The determination of the communication performance characteristic level (which may include measurements of latency and / or throughput) may use communication capability information from data exchange between the plurality of system units.

[0057] Hardware processor 702 can execute instructions 710 to associate the communication architecture between each of the multiple system components with each of the multiple workflow code execution components, based on the communication performance characteristic level associated with each workflow code execution component. This allows the mapping to be dynamically adjusted according to the running workload by analyzing workflow communication. More specifically, the mapping can be dynamically changed based on data communication patterns (e.g., using communication performance characteristics such as latency and / or throughput) and / or data locality. In an example implementation in this regard, the communication architecture associated between a pair of system components and with the workflow code execution component having the highest communication performance characteristic level is given the highest priority among the remaining communication architectures.

[0058] In some examples, the communication architecture between two system components out of multiple system components is correlated based on a comparison of the communication performance characteristic levels between two workflow code execution components within the workflow code execution component.

[0059] In some examples, multiple system components do not use a centralized server to communicate with each other.

[0060] In some examples, multiple system components include a CPU and at least one accelerator.

[0061] In some examples, multiple system components include multiple accelerators.

[0062] In some examples, communication architectures include memory-mapped communication architectures, PCIe (PCIe) communication architectures, Ethernet communication architectures (such as Slingshot, Compute Fast Link (CXL), NVLink, etc.), or Infiniband communication architectures.

[0063] Figure 8 A block diagram of an example computer system 800 is depicted, in which the various aspects described herein can be implemented. The computer system 800 can be used to execute machine-readable instructions to perform the processes described herein.

[0064] Computer system 800 includes a bus 802 or other communication mechanism for communicating information, and one or more hardware processors 804 coupled to the bus 802 for processing information. The one or more hardware processors 804 may be, for example, one or more general-purpose microprocessors.

[0065] Computer system 800 also includes main memory 806, such as random access memory (RAM), cache, and / or other dynamic storage devices, coupled to bus 802 for storing information and instructions to be executed by processor 804. Main memory 806 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 804. When these instructions are stored in storage media accessible to processor 804, computer system 800 becomes a dedicated machine customized to perform the operations specified in the instructions.

[0066] The computer system 800 also includes a read-only memory (ROM) 808 or other static storage device coupled to the bus 802 for storing static information and instructions for the processor 804. Storage devices 810, such as disks, optical discs, or USB thumb drives (flash drives), are provided and coupled to the bus 802 for storing information and instructions.

[0067] Computer system 800 can be coupled to display 812, such as a liquid crystal display (LCD) (or touchscreen), via bus 802 for displaying information to a computer user. Input device 814, including alphanumeric and other keys, is coupled to bus 802 for communicating information and command selections to processor 804. Another type of user input device is cursor control 816, such as a mouse, trackball, or arrow keys, for transmitting directional information and command selections to processor 804 and for controlling cursor movement on display 812. In some examples, the same directional information and command selections as cursor control can be implemented by receiving touches on a touchscreen without a cursor.

[0068] The computing system 800 may include a user interface module to implement a GUI, which may be stored as executable software code executed by one or more computing devices in a mass storage device. For example, this module and other modules may include components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, programs, subroutine routines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

[0069] Generally, terms such as "component," "engine," "system," "database," and "data storage," as used herein, can refer to logic embodied in hardware or firmware, or a set of software instructions that may have entry and exit points written in a programming language (e.g., Java, C, or C++). Software components can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language (e.g., BASIC, Perl, or Python). It should be understood that software components can be invoked from other components or themselves, and / or can be invoked in response to detected events or interrupts. Software components configured to execute on a computing device can be provided on computer-readable media, such as optical discs, digital video discs, flash drives, magnetic disks, or any other tangible media, or as digital downloads (and can be stored raw in a compressed or installable format that requires installation, decompression, or decryption before execution). This software code can be stored, in part or in whole, on the memory device of the computing device in which it is executing, so that it can be executed by that computing device. Software instructions can be embedded in firmware such as EPROM. It should also be understood that hardware components may consist of connected logic units such as gates and flip-flops, and / or may consist of programmable units such as programmable gate arrays or processors.

[0070] Computer system 800 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that combine with or program the computer system to make it a dedicated machine. According to one example, the techniques of this invention are executed by computer system 800 in response to one or more processors 804 executing one or more sequences of one or more instructions contained in main memory 806. These instructions may be read into main memory 806 from another storage medium, such as storage device 810. Execution of the sequence of instructions contained in main memory 806 causes one or more processors 804 to perform the processing steps described herein. In alternative examples, hardwired circuitry may be used in place of or in combination with software instructions.

[0071] As used herein, “non-transitory media” and similar terms refer to any medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such non-transitory media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 810. Volatile media include dynamic memory, such as main memory 806. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, fast erasable programmable read-only memory (FLASH-EPROM), NVRAM, any other memory chips or cassettes, and their network versions.

[0072] Non-transient media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the information transmission between non-transient media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including conductors that include bus 802. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0073] Computer system 800 also includes a communication interface 818 coupled to bus 802. Communication interface 818 provides bidirectional data communication between one or more network links connected to one or more local area networks (LANs). For example, communication interface 818 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 818 may be a LAN card to provide data communication connectivity with a compatible LAN (or a WAN component communicating with a WAN). Wireless links may also be implemented. In any such implementation, communication interface 818 transmits and receives electronic, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0074] Network links typically provide data communication to other data devices via one or more networks. For example, a network link can provide a connection to a host computer or a data device operated by an Internet Service Provider (ISP) via a local area network (LAN). ISPs, in turn, provide data communication services via a global packet data communication network (now commonly referred to as the "Internet"). Both LANs and the Internet use electronic, electromagnetic, or optical signals to carry digital data streams. Signals through various networks, as well as signals on network links and through communication interface 818, are example forms of transmission media that transmit digital data to and from computer system 800.

[0075] Computer system 800 can send messages and receive data, including program code, through one or more networks, network links, and communication interfaces 818. In the Internet example, the server can transmit code requested by the application through the Internet, ISP, local area network, and communication interface 818.

[0076] The received code can be executed by processor 804 upon receipt and / or stored in storage device 810 or other non-volatile storage device for later execution.

[0077] Each process, method, and algorithm described in the preceding chapters can be embodied in a code component executed by one or more computer systems or computer processors including computer hardware, and the code component executed by one or more computer systems or computer processors including computer hardware can be fully or partially automated. One or more computer systems or computer processors can also operate to support the execution of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). Processes and algorithms can be implemented, partially or wholly, in dedicated circuitry. The various features and processes described above can be used independently of each other or can be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order; associated blocks or states can be executed in other suitable orders, or can be executed in parallel, or in some other way. Blocks or states can be added to or removed from the disclosed examples. The execution of certain operations or processes can be distributed among computer systems or computer processors, not residing only within a single machine, but deployed across multiple machines.

[0078] As used herein, circuits can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be implemented to form a circuit. In implementation, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be shared partially or wholly among one or more circuits. Even if various features or functional elements may be described separately or claimed as separate circuits, these features and functions can be shared among one or more general-purpose circuits, and such description does not require or imply the need for separate circuits to implement these features or functions. If the circuit is implemented wholly or partially using software, the software can be implemented to operate in conjunction with a computing or processing system (e.g., computer system 800) capable of performing the functions described therein.

[0079] As used herein, the term “or” can be interpreted as inclusive or exclusive. Furthermore, descriptions of resources, operations, or structures in the singular form should not be construed as excluding the plural form. Conditional language, such as “can”, “could”, “might”, “may”, unless explicitly stated otherwise or otherwise understood in the context in which they are used, is generally intended to convey that certain features, elements, and / or steps are included in some examples of this disclosure while not included in others.

[0080] The terms and phrases used herein, and their variations thereof, should be interpreted as open-ended rather than restrictive, unless otherwise expressly stated. Adjectives such as “customary,” “traditional,” “regular,” “standard,” “known,” and similar terms should not be interpreted as limiting the described items to those available up to a given time period, but should be understood to include customary, traditional, regular, or standard techniques that may be available or known now or in the future. In some cases, the presence of extended words and phrases such as “one or more,” “at least,” “but not limited to,” or other similar phrases should not be interpreted as implying a narrower scope intended or required where such extended phrases may not exist.

Claims

1. A non-transitory machine-readable medium storing instructions that cause one or more processors of an electronic system to perform operations, the operations including: Map multiple workflow code execution components to multiple system components respectively; Determine the communication performance characteristic level associated with each of the plurality of workflow code execution components; as well as Based on the communication performance characteristic level associated with each of the plurality of workflow code execution components, the communication architecture between each of the plurality of system components is associated with each of the plurality of workflow code execution components, wherein the association of the communication architecture between two of the plurality of system components is based on a comparison of the communication performance characteristic levels between the two workflow code execution components.

2. The non-transient machine-readable medium according to claim 1, wherein the communication performance characteristic level includes delay measurements and / or throughput measurements.

3. The non-transitory machine-readable medium of claim 2, wherein the communication architecture associated with a pair of system components and with a workflow code execution component having the highest level of communication performance characteristics is given the highest priority among the remaining communication architectures.

4. The non-transitory machine-readable medium of claim 1, wherein the plurality of system components do not use a centralized server to communicate with each other.

5. The non-transitory machine-readable medium of claim 1, wherein the plurality of system components includes a CPU and at least one accelerator.

6. The non-transitory machine-readable medium of claim 1, wherein the plurality of system components includes a plurality of accelerators.

7. The non-transitory machine-readable medium of claim 1, wherein the communication architecture includes a memory-mapped communication architecture, a peripheral component interconnect fast (PCIe) communication architecture, an Ethernet communication architecture, or an unlimited bandwidth communication architecture.

8. The non-transitory machine-readable medium of claim 1, wherein communication capability information obtained from data exchange between each of the plurality of system components is used to determine the level of communication performance characteristics.

9. A method comprising: Provides a communication architecture between the first system component and the second system component; Identify the workflow code execution component that has the highest level of communication performance characteristics among other workflow code execution components; as well as Associating the communication architecture with the workflow code execution component having the highest communication performance characteristic level, wherein the method further includes assigning the highest priority to the communication architecture among other communication architectures provided between other system components.

10. The method of claim 9, wherein the communication performance characteristic level includes latency measurements and / or throughput measurements.

11. The method of claim 9, wherein the first system component and the second system component do not use a centralized server to communicate with each other.

12. The method of claim 9, wherein the first system component and the second system component comprise a plurality of accelerators.

13. The method of claim 9, wherein the identification of the workflow code execution component having the maximum communication performance characteristic level is performed using communication capability information provided by at least one of the first system component and the second system component.

14. A communication architecture system, comprising: First system component; Second system component; Third system components; Multiple workflow code execution components; processor; and A memory configured to store instructions, the instructions being executed by the processor, causing the processor to perform the following steps: A first communication architecture is provided between the first system component and the second system component; A second communication architecture is provided between the second system component and the third system component; Determine which of the first and second communication architectures has the highest priority; Compare the execution speeds of the multiple workflow code execution components; The comparison step determines which of the plurality of workflow code execution components has the highest execution speed; and The workflow code execution component with the maximum execution speed is associated with the first or second communication architecture that is determined to have the highest priority.

15. The communication architecture system of claim 14, wherein the comparison of the execution speed of the plurality of workflow code execution units includes a comparison of latency measurements and / or throughput measurements.

16. The communication architecture system of claim 14, wherein the first system component, the second system component, and the third system component do not use a centralized server to communicate with each other.

17. The communication architecture system of claim 14, wherein the first system component, the second system component and the third system component include a CPU and a plurality of accelerators.

18. The communication architecture system of claim 14, wherein communication capability information provided by at least one of the first system component, the second system component, and the third system component is used to identify the workflow code execution component having the maximum execution speed.

Images