Hybrid Hardware-Software Coherent Framework

Abstract

The accelerator device (140) shares the same coherent domain as the hardware elements of the host computing device (105). The mix of hardware and software coherence reduces the overhead of managing data when large chunks of data are moved from the host to the accelerator device. An accelerator application (125) running on the host identifies a data set (130) to be sent to the accelerator device to be processed. The accelerator application transfers ownership from the host's home agent (135) to the accelerator device. Then, the slave agent (155) can take ownership of the data. Consequently, any memory operation requests received from the accelerator device's request agent (145) allow the slave agent to access the data set in local memory (160) through the slave agent without obtaining permission from the host's home agent.

Description

Technology Field

[0001] Examples of the present disclosure generally relate to an accelerator that uses a collaborative hardware-software coherent framework to transmit data between a host and an accelerator device. Background Technology

[0002] In a conventional I / O model, the host computing system interfaces with peripheral I / O devices (e.g., accelerators) when executing accelerator tasks or functions using custom I / O device drivers unique to the peripheral I / O devices. A disadvantage of this model is that all accelerator-memory interactions must pass through the home agent node of the main CPU-memory complex, sometimes referred to as the server host. Since this model uses hardware enforced coherency, all accelerator-memory interactions are serviced and tracked by the home agent node, regardless of whether the accelerator-memory data footprint is tens of times larger than the cacheline (MB, GB, or even TB), even if the processor (producer) on the server host can only access a fraction of the data footprint where the accelerator has completed the computations. Another disadvantage is that all memory accessed and cached by the accelerator is attached memory of the remote server host. An additional disadvantage of this model is that PCIe I / O-based accelerators cannot cache memory. Accelerator-memory interactions occur solely due to the server host attached memory and the disadvantages of latency, bandwidth, and protocol messaging overhead associated with that approach.

[0003] Meanwhile, the hardware cache-coherent shared-memory multiprocessor paradigm utilizes a generic ISA (instruction set architecture)-independent model that interfaces with execution tasks or functions on multiprocessor CPUs. The interfaced generic ISA-independent (e.g., C-code) model scales according to both the number of processing units and the amount of shared memory available to those processing units. Typically, peripheral I / O devices could not take advantage of the benefits of the coherent paradigm used by CPUs running on the host computing system.

[0004] Techniques for transferring ownership of a data set to an accelerator device are described. One example is a computing system, which includes a host comprising a processing unit, a home agent (HA), and an accelerator application, and an accelerator device communicably coupled to the host, wherein the accelerator device includes a request agent (RA), a slave agent (SA), and local memory configured to perform at least one accelerator function, wherein the local memory is part of the same coherent domain as the processing unit of the host. The accelerator application is configured to transfer ownership of a data set from the HA to the accelerator device, and the accelerator device is configured to store the latest copy of the data set in local memory, service a memory request from the RA using the SA, and transfer ownership of the data set back to the HA.

[0005] In some embodiments, the accelerator application is configured to identify a data set, and the data set is a sub-part of a memory block or memory page.

[0006] In some embodiments, after ownership of the data set is transferred from the HA to the accelerator device, the remainder of the memory block or memory page continues to be homed by the HA.

[0007] In some embodiments, servicing memory requests from RA using SA is performed without receiving permission from HA.

[0008] In some embodiments, the processing units include a cache, and the cache stores a modified version of the data set.

[0009] In some embodiments, the accelerator device is configured to send a flush command to the HA instructing the processing unit to flush the cache so that a modified version of the data set is moved to local memory after ownership of the data set is transferred to the accelerator device.

[0010] In some embodiments, the processing units are configured to either invalidate the modified version of the data set in the cache in response to a flush command or to retain a cached clean copy of the modified version of the data set for future low-latency re-reference.

[0011] In some embodiments, the accelerator application is configured to send a flush command to the HA instructing the processing unit to flush the cache so that a modified version of the data set is moved to local memory before ownership of the data set is transferred to the accelerator device.

[0012] In some embodiments, the accelerator application is configured to send an invalidate command to the HA that commands the processing unit to invalidate the cache before ownership of the data set is transferred to the accelerator device.

[0013] An example described herein is an accelerator device, which comprises an RA including a compute engine configured to perform at least one accelerator function, an SA including a memory controller, and memory. The RA is configured to receive ownership of a data set from the HA of a host coupled to the accelerator device to the SA, wherein ownership is transferred by a software application. The memory controller is configured to service a request sent by the compute engine to access the data set stored in memory once ownership is transferred to the SA. The RA is configured to transfer ownership back from the SA to the HA.

[0014] In some embodiments, the data set is a sub-part of a memory block or memory page, and after ownership of the data set is transferred from the HA to the SA, the remainder of the memory block or memory page continues to be homed by the HA, and the service of requests is performed without receiving permission from the HA.

[0015] In some embodiments, the accelerator device is configured to communicate with the host using a coherent interconnection protocol to extend the coherent domain of the host to include memory and SA in the accelerator device.

[0016] One example described herein is a method comprising the steps of transferring ownership of a data set from the HA of a host to an accelerator device—whereby the accelerator device is communicably coupled to the host and the coherent domain of the host is extended to the accelerator device—moving the data set to the local memory of the accelerator device, servicing a memory request from the RA of the accelerator device using the SA of the accelerator device, and transferring ownership from the accelerator device back to the HA.

[0017] In some embodiments, the method includes the step of identifying a data set using an accelerator application running on a host, wherein the data set is a sub-part of a memory block or memory page, and after ownership of the data set is transferred from the HA to the accelerator device, the remainder of the memory block or memory page continues to be homed by the HA.

[0018] In some embodiments, the method includes the step of invalidating a modified data set of the cache in response to a cache maintenance operation (CMO) issued by the accelerator device after ownership has been transferred to the accelerator device. Brief explanation of the drawing

[0019] In order to allow the features mentioned above to be understood in detail, a more specific description of the briefly summarized above may be provided with reference to exemplary implementations, some of which are illustrated in the attached drawings. However, it should be noted that the attached drawings illustrate only ordinary exemplary implementations and should not be construed as limiting the scope of such implementations. FIG. 1 is a block diagram of a host coupled to an accelerator device according to one example. FIG. 2 is a block diagram of a host coupled to an accelerator device according to one example. Figure 3 is a flowchart for using a collaborative hardware-software coherent framework according to one example. Figure 4 is a flowchart for using a collaborative hardware-software coherent framework according to one example. FIG. 5 illustrates an implementation of a field programmable gate array of a programmable IC according to one example. Specific details for implementing the invention

[0020] Various features are described below with reference to the drawings. It should be noted that the drawings may or may not be drawn to actual scale, and that elements of similar structures or functions are represented by similar reference numbers throughout the drawings. It should be noted that the drawings are intended only to facilitate the description of features. They are not intended to be a comprehensive description of the details or to limit the scope of the claims. Additionally, the illustrated examples are not required to have all the aspects or advantages illustrated. An aspect or advantage described with a particular example is not necessarily limited to that example and may be practiced in any other examples that are not so illustrated or so explicitly described.

[0021] The examples herein describe accelerator devices (e.g., peripheral I / O devices or accelerator-attached memory) that share the same coherent domain as the hardware elements of the host computing device. Consequently, compute resources in the coherent domain of the accelerator device can communicate with the host in a manner similar to CPU-to-CPU communication on the host. This means that compute resources can utilize coherence-type features such as direct communication (without address translation), more efficient memory usage, and NUMA (non-uniform memory access) awareness. Coherent memory relies on fine-grained memory management, where the home agent node manages memory at the cache line level.

[0022] However, accelerator tasks typically handle large chunks of memory that can be better managed using software coherence rather than hardware coherence, where the host's home agent manages the data processed by the accelerator device. For example, a disadvantage of the hardware coherence model is that all accelerator-memory interactions still pass through the server host first. This model uses hardware-forced coherence regardless of whether the accelerator-memory data footprint is tens of times larger than the cache line (MB, GB, or even TB) and regardless of whether the memory is accelerator-attached; all accelerator-memory interactions are still serviced and tracked by the host first, even if the processor (producer) can only access a portion of the data footprint where the accelerator has completed computations. In contrast, software coherence is a case where an accelerator device manages data stored in its memory, similar to a conventional host-I / O model, where the host transmits large chunks of data to the I / O device, the device processes the data, and the I / O device notifies the host when the processed data is ready. Meanwhile, the software coherence model guarantees that the host does not access the data until the I / O accelerator prepares the processed data.

[0023] The embodiments of the present invention describe a combination of hardware and software coherence to create a hybrid hardware-software framework, which reduces the overhead of managing data when moving large chunks of data from a host to an accelerator device when the host and the accelerator are in the same coherent domain. In one embodiment, an accelerator application running on a host identifies a data set to be sent to an accelerator device to be processed. The accelerator application notifies the accelerator device via software coherent methods by modifying metadata regarding the data or by modifying control data structures accessed by the accelerator device. In response, the accelerator device sends a request to the host using hardware coherent techniques to request the most up-to-date version of the data set and to ensure that no cached copies of the data set remain on the host. Then, using hardware coherent techniques, the host ensures that the most up-to-date version of the data is stored in the memory of the accelerator device and invalidates all copies of the data set in the CPU cache. The request agent of the accelerator device initiates the processing of a data set in local memory via a slave agent according to the acceleration application. In one embodiment, the request agent may cache subsets of local memory and process the cached copies. Software coherence ensures that any memory operation requests received from the request agent of the accelerator device allow the slave agent to access the data set in local memory via the slave agent without obtaining permission from the host. For example, the slave agent does not need to check with the home agent whether a more recent version of the data set exists or whether a cached copy of the data to be processed by the accelerator device remains.When the accelerator device has processed the data set, the accelerator device flushes cached subsets, updates local memory, and transfers ownership back to the host's home agent via software coherent methods, for example, by modifying control data structures or metadata accessed by host processors. This resets the host as an enforcer of hardware coherence for the data set with the rest of the system, similar to when work is completed in a conventional host I / O model where the accelerator device is in a different domain from the host. Therefore, by being in the same coherence domain and relying on hardware coherence enforcement, the accelerator device and the host avoid the high overhead that occurs when transferring ownership of large chunks of data to the accelerator device's local memory via software methods. By being in the same coherence domain, the accelerator device and the host can also avoid the high overhead of moving processed data from the accelerator device's local memory back to the host.

[0024] FIG. 1 is a block diagram of a computing system (100) comprising a host (105) coupled to an accelerator device (140) according to one example. The accelerator device (140) may be any device (e.g., peripheral I / O device) that performs tasks issued by the host (105). In one embodiment, the host (105) is communicably coupled to the accelerator device (140) using a PCIe connection. In another embodiment, the accelerator device (140) is part of a host-attached memory rather than a separate I / O device. The host (105) may represent a single computer (e.g., a server) or a number of interconnected physical computing systems. In any case, the host (105) includes a number of CPUs (110), memory (115), and a home agent (135).

[0025] Memory (115) includes an operating system (OS) (120) which may be any OS capable of performing the functions described herein. In one embodiment, the OS (120) (or hypervisor or kernel) establishes a cache-coherent shared-memory multiprocessor paradigm for CPUs (110) and memory (115). In one embodiment, the CPUs (110) and memory (115) are OS managed (or kernel / hypervisor managed) to form coherent domains following the cache-coherent shared-memory multiprocessor paradigm.

[0026] In the embodiments of the present invention, the shared-memory multiprocessor paradigm is available to the accelerator device (140) with all performance advantages, software flexibility, and reduced overhead of the paradigm. Additionally, adding the compute resources of the accelerator device (140) to the same coherent domain as the CPUs (110) and memory (115) allows for a general ISA-independent development environment.

[0027] In one embodiment, the accelerator device (140) and the host (105) use a coherent interconnect protocol to extend the coherent domain of the host (105) to the accelerator device (140). For example, the accelerator device (140) may use CCIX (cache coherent interconnect for accelerators) to extend the coherent domain within the device (140). CCIX is a high-performance chip-to-chip interconnect architecture that provides a cache coherent framework for heterogeneous system architectures. CCIX provides kernel managed semantics to the accelerator device (140). Cache coherence is automatically maintained between the CPU(s) (110) on the host (105) and various other accelerators of the system that may be placed on any number of peripheral I / O devices.

[0028] However, other coherent interconnect protocols other than CCIX, such as QuickPath Interconnect (QPI), Omni-Path, Infinity Fabric, NVLink, or OpenCAPI, may be used to extend the coherent domain of the host (105) to include compute resources in the accelerator device (140). That is, the accelerator device (140) may be customized to support any type of coherent interconnect protocol that facilitates the formation of a coherent domain including compute resources in the accelerator device (140).

[0029] In one embodiment, the HA (135) executes coherency actions for the memory of the system (100) (both the memory (115) of the host (105) and the memory (160) of the accelerator device (140). In a coherent system, memory address ranges are homed by a designated HA, which provides the benefit of rapid coherency resolution for cache lines accessed by the CPUs (110) of the host (105) as well as the request agent (145) of the accelerator device (140). The HA (135) ensures that the most recent versions of data, which may be stored simultaneously in different locations (or caches), are retrieved when a request to read or modify said data is received. Such requests may be sent from the request agent (e.g., request agent (145)) of the attached accelerator device (140) or from the CPU (110). Although one HA (135) is shown, the computing system (100) may have any number of HAs when each HA serves for the home as different memory address ranges.

[0030] The accelerator device (140) may be various different types of peripheral devices, such as a pluggable card (plugged into an expansion slot of the host (105)), a system on a chip (SoC), a graphics processing unit (GPU), a field programmable gate array (FPGA), etc. For example, the accelerator device (140) may include programmable logic (e.g., a programmable logic array) or may not include any programmable logic but instead include only a hardened circuit (which may be software programmable).

[0031] The accelerator device (140) includes a request agent (RA) (145), a slave agent (SA) (155), and memory (160) (also referred to as local memory). In one embodiment, the RA (145) is a compute engine optimized to perform a specific computation. In one embodiment, the RA (145) is hardware, firmware, or a combination thereof. For example, the RA (145) may be one or more compute engines implemented as programmable logic. In another example, the RA (145) may be a compute engine formed from an enhanced circuit (e.g., a core of a GPU or other ASIC). In any case, the RA (145) performs accelerator functions (150) for the accelerator device. These functions (150) may be part of a machine learning accelerator, a cryptographic accelerator, a graphics accelerator, a search accelerator, a decompression / compression accelerator, etc. In one embodiment, the accelerator functions (150) are implemented as programmable logic (e.g., a programmable logic array) in the accelerator device (140). In another embodiment, the accelerator functions (150) are implemented as enhanced logic (which may be software configurable), such as processing cores or engines.

[0032] Generally, the compute engine or engines of the RA (145) perform the accelerator functions (150) faster or more efficiently than when the same functions are performed using one or more of the CPUs (110) of the host (105). Although one accelerator function (150) and one accelerator application (125) are illustrated, the computing system (100) may have any number of accelerator functions, each accelerator function is to process a different data set in memory.

[0033] In one embodiment, the SA (155) is a memory controller (which may be a hardware circuit) that services requests to read data from the memory (160) or to write data to the memory (160). The RA (145) ensures that coherence is maintained when data memory operations are performed from the RA (145) to the memory (160). For example, if the RA (145) wants to read data from the memory (160), the RA (145) may first send a request to the HA (135) to determine whether a more recent version of the data is available elsewhere in the system (100). Additionally, while the memory (160) is depicted within the accelerator device (140), in other embodiments, the memory (160) may be on a chip (e.g., an attached memory) separate from the integrated circuit forming the accelerator device (140).

[0034] As described in more detail below, SA (155) has different functions depending on the current phase or configuration of the system (100). In one phase, HA (135) is the sole agent for the data stored in memory (160), and SA (155) functions like a typical memory controller in a coherent system. However, in another phase, software coherence has granted ownership of the data stored in local memory (160) to SA (155), and RA (145) does not need to ask HA (135) for permission when processing the data stored in memory (160). For example, when RA (145) wants to read data from memory (160) to perform accelerator functions (150), in the second phase, SA (155) can provide the data without first checking HA (135) to determine whether there is a newer version of the data somewhere in the system (100) (e.g., in memory (115) or in an in-chip cache within one of the CPUs (110).

[0035] Although RA (145) is exemplified as being separate from memory (160), in one embodiment, RA (145) may be a compute engine embedded within memory (160) and performing in-memory computation. In such a case, the accelerator device (140) may not have physically distinct RA (145) and SA (155) as illustrated in FIG. 1, but the accelerator locality may be a memory locality having logically distinct RA (145) and SA (155) that perform in-memory computation.

[0036] FIG. 2 is a block diagram of a host (105) coupled to an accelerator device (140) according to one example. Unlike a conventional I / O model, the memory (160) and processing elements (e.g., RA (145) and SA (155)) of the accelerator device (140) are in the same coherent domain as the CPU (110) and memory of the host (105). Thus, the HA (135) of the host (105) ensures that the data stored in the host (105) and the accelerator device (140) is coherently stored, so that a request for a memory operation, whether originating from the host (105) or the accelerator device (140), receives the most recent version of the data regardless of whether the data is stored in the memory of the host (105) or the accelerator device (140).

[0037] In some applications, the bottleneck during data processing arises not from the time it takes to process the data, but from the movement of the data to the compute unit processing it. In other words, moving data to and from a processor or compute engine limits the speed at which operations can be performed, rather than bottlenecking the processor or compute engine itself. This situation often occurs when accelerators are used, as they possess specialized compute engines (e.g., accelerator engines) capable of performing certain types of functions exceptionally quickly. Often, what limits the efficiency of accelerator usage is not the compute engines, but the system's ability to move data to the accelerator for processing. Situations where moving data to a compute engine limits the time required to complete a task are referred to as computational memory bottlenecks. Moving the compute engine closer to the data can help mitigate this problem.

[0038] In the first embodiment, a conventional I / O data sharing model may be used when large chunks of data are moved from the host (105) to the accelerator device (140). However, the host (105) and the accelerator device (140) are no longer in the same coherence domain. Consequently, the I / O-based accelerator device cannot cache memory, which further separates the compute engine (e.g., RA (145)) from the data it needs to process. Instead of using hardware coherence, software coherence is used when a software application (e.g., accelerator application (125)) sets up with HA (135) any data it manages and intends to send to the accelerator device (140) (e.g., data set (130)). Once the accelerator device (140) completes data processing, the accelerator application (125) can take ownership of the data again and maintain (or reset) the hardware coherence with the HA (135) and the data.

[0039] In a second embodiment, the host (105) and the accelerator device (140) may share the same coherent domain so that data can be cached in the accelerator device (140) to help alleviate computational memory bottlenecks. For example, memory may be attached to the host (105) (e.g., host-attached memory), while data may be stored in the cache of the RA (145). However, a disadvantage of this model is that all accelerator-memory interactions pass through the HA (135) in the main CPU-memory complex of the host (105). This embodiment also uses hardware-forced coherence, so all accelerator-memory interactions are serviced and tracked by the HA (135), regardless of whether the accelerator-memory data footprint is tens of times larger (MB, GB, or even TB) than the cache line. In other words, the HA (135) must track all data sent to the RA (145) cache, even if the CPU (110A) can only access a portion of the data where the accelerator device (140) has completed computation. Another disadvantage is that all memory accessed and cached by the accelerator device (140) is remote host-attached memory.

[0040] In the third embodiment, the memory storing data processed by the accelerator device (140) is attached to (or inside) the accelerator device (140) (e.g., memory (160)), but is homed by the HA (135) of the host (105). In this embodiment, the disadvantages of the second embodiment also apply in that all accelerator-memory interactions still pass through the HA (135) of the host (105). However, unlike the previous embodiment, once ownership of the cache line is acquired by the accelerator device (140), updates to the data set (130) can occur locally in the memory (160). This embodiment also uses hardware-forced coherence, so the other disadvantage of the second embodiment also applies in that all accelerator-memory interactions continue to be serviced first and tracked by the HA (135), regardless of whether the accelerator-memory data footprint is several tens of times (MB, GB, or even TB) larger than the cache line and regardless of whether the memory is accelerator-attached.

[0041] Instead of the three embodiments above, a collaborative framework may be used when the benefits of the first, second, and third embodiments above are integrated into a system while avoiding their disadvantages. In addition to the benefits above, the collaborative framework has additional advantages. In one embodiment, memory is continuously homed by the HA (135) of the host (105), so the benefit of rapid coherence analysis for cache lines accessed by the CPUs (110) of the host (105) is continued. Also, unlike requiring all accelerator accesses to memory to be tracked by software coherence at a coarse level as in the first embodiment, or requiring all accelerator accesses to memory to have fine-grain tracking as in the second and third embodiments, in the collaborative framework, only a fine-grain subset of accelerator-memory interactions first pass through the HA (135) to claim ownership.

[0042] Regarding another advantage of the collaboration framework, unlike software coherence where all memory pages or entire memory blocks can be flushed from CPU caches (e.g., cache (210)), for a subset of accelerator-memory interactions going to HA (135), only the parts cached in the host CPU (110) are snooped from the CPU cache (210). Accelerator-memory interactions that do not touch the contents of the CPU cache (210) can be kept cached to provide low-latency access to the corresponding data to the CPUs (110) (and their cores (205)).

[0043] Regarding another advantage of the collaboration framework, unlike the second and third embodiments where accelerator-memory interactions occur only in host-attached memory and the disadvantages of latency, bandwidth, and protocol messaging overhead of the approach, using the collaboration framework, accelerator-memory interactions can occur in accelerator-attached memory (160), where the accelerator location is near-memory or in-memory, providing a low-latency, high-bandwidth dedicated path and minimizing protocol messaging overhead resulting from subsequent private accelerator-memory interactions between RA (145) and SA (155).

[0044] Regarding another advantage of the collaboration framework, unlike the third embodiment where interactions with memory require home node ownership even though the accelerator location is near or within memory, in the collaboration framework, once the accelerator device (140) claims ownership of the data set (130) on which computation is being performed, all subsequent computational memory actions occur without interactions with the HA (135) until ownership is returned to the HA (135).

[0045] The following examples describe different techniques for achieving the above benefits using a collaboration framework. However, the collaboration framework is not limited to these benefits. Some implementations of the framework may have fewer benefits than those listed above, while others may have different benefits not listed.

[0046] FIG. 3 is a flowchart of a method (300) for using a collaborative hardware-software coherent framework, according to one example. In block (305), the accelerator application identifies a data set to be transmitted to the accelerator for processing. In one embodiment, the accelerator application is a software application that performs a task to benefit from the accelerator device. For example, the accelerator application may be a machine learning application in which the accelerator device includes one or more machine learning engines, a video application in which the accelerator device performs graphics processing, a search engine in which the accelerator device searches for specific words or phrases, or a security application in which the accelerator device performs encryption / decryption algorithms on data.

[0047] In any case, the accelerator device can recognize a collaboration framework that allows the transfer of data ownership from the HA on the host to the SA on the accelerator device while the host and the accelerator device share the same coherence domain. Thus, the accelerator device can identify the data set it wishes to transmit to the accelerator device as part of the collaboration framework.

[0048] In block (310), an accelerator application (e.g., or some other software application) transfers ownership of a dataset from a server host to an accelerator device. In one embodiment, the accelerator application updates a flag or metadata monitored by the accelerator device. Updating this transfers ownership of the dataset from the HA to the accelerator device (e.g., the RA of the accelerator device). Through this transfer of ownership, the hardware of the accelerator device knows that it can operate on the dataset without worrying about the HA of the host (since the accelerator device now owns the dataset).

[0049] In one embodiment, the accelerator application transfers ownership only to a subset of data on which computational memory actions are to be performed. That is, the accelerator application can identify only the data that the accelerator device will process as part of its accelerator functions. Thus, the accelerator application on the host can transfer ownership only to the data that the accelerator device will process. This is an advantage over conventional I / O models and software coherence, where all pages or entire memory blocks are flushed from CPU caches rather than just the required data set (which may be a subportion of these pages and memory blocks).

[0050] In block (315), the accelerator device (or more specifically, the RA) exercises its ownership of the data set by requesting the latest copy of the data set from the server host. Doing so moves the data set into the accelerator device's local memory. In one embodiment, the data set may be ensured to remain in the accelerator device's local memory as part of maintaining hardware coherence between the host and the accelerator device. In such a case, the data set identified in block (305) may have already been stored in the accelerator's local memory before the accelerator application transferred ownership in block (310). However, in another embodiment, the RA (or accelerator device) may use flush and / or invalidate commands to ensure that the data set is moved into the accelerator device's local memory and / or that no cached copies remain. Various techniques to ensure that the accelerator device's local memory holds the most recent version of the data set are described in more detail later in FIG. 4.

[0051] In block (320), RA services memory requests using SA. When SA services memory requests from an accelerator device, SA can service these requests without first checking with HA, for example, whether a newer version of the data is available elsewhere in the system. That is, the data set is now homed by the accelerator device.

[0052] In one embodiment, while the data set is homed by the accelerator device, the HA does not track any modifications to the data set. This reduces system overhead because the RA can read and modify the data set without the SA first requiring permission from the HA. As discussed above, the HA tracks data by cache line, which means that each cache line of the data set must be checked by the HA before servicing memory operations. However, since the SA now homes the data set, this function is performed solely by the SA without assistance from the host's HA.

[0053] Additionally, software coherence guarantees that while the data set is being processed by the RA, the host CPU can only read the data set and cannot write to it unless the RA indicates that processing is complete. Software coherence also guarantees that if the CPU reads data that has not yet been updated, the CPU must not derive any meaning from "stale" values ​​from the read when performing that read. Only after the accelerator returns ownership to the software / HA can the CPUs derive meaning from values ​​in the updated data.

[0054] In block (325), RA determines whether he has finished the data set. If RA has not finished performing his accelerator functions on the data set, method (300) is repeated to 320, where RA continues to send additional memory requests for parts of the data set to SA. However, once RA has finished processing the data set, method (300) proceeds to block (330), where RA transfers ownership of the data set back to HA. Thus, in block (330), it is the hardware of the accelerator device, not the software transferring ownership as performed in block (310), that sets software-accessible metadata indicating to the accelerator application that the accelerator device has finished with the data set. In one embodiment, RA updates the metadata when the producer-consumer model requires that RA notify the CPU when he has finished performing the accelerator functions on the data set. HA can ensure that the hardware coherence of the data (modified) is maintained (or reset) with on-host entities such as CPU caches and host-attached memory. In other words, software (e.g., an accelerator application) uses HA to implicitly exercise ownership of the data set. The accelerator application knows HA only as a coherence enforcer.

[0055] In one embodiment, transferring ownership of a data set from HA to SA and back to HA as illustrated in method (400) is different from software coherence used in conventional I / O models, but can be considered as a type of software coherence because it is not required to identify the target data set and flush / invalidate the host caches (e.g., CPU caches).

[0056] FIG. 4 is a flowchart for a method (400) for using a collaborative hardware-software coherent method, according to one example. Generally, the method (400) describes different techniques for moving data from various memory elements of a host to the local memory of an accelerator device so that the HA can transfer ownership of the data to the SA, as described in the method (300).

[0057] In block (405), the CPU caches at least a portion of the data set. That is, a portion of the data set may be stored in the CPU's cache before the accelerator application transfers ownership of the data set. In block (410), the CPU modifies a portion of the data set stored in its cache. That is, a portion of the data set may be stored in various caches and memory of the host before the accelerator application transfers ownership in block (310) of method (300). The data set stored in the various caches may be duplicate copies of memory in the host or updates to the data set as a result of software (e.g., the accelerator application) implicitly exercising ownership of the data set using HA. Thus, before transferring ownership of the data set from the accelerator device to the SA, the system requires a technique to ensure that the most recent version of the data set is stored in the accelerator attachment memory (i.e., local memory).

[0058] In block (415), the method (400) is subdivided depending on whether software or hardware is used to transmit a modified data set to the accelerator device. That is, the task of ensuring that the accelerator device has the most up-to-date data set can be performed by the hardware (e.g., RA) or software (e.g., accelerator application) of the accelerator device.

[0059] Assuming this task is assigned to software, the method (400) proceeds to a block (420) that ensures the host's accelerator application transmits the modified data to the accelerator device. In one embodiment, the block (420) is performed before the accelerator device transfers ownership of the data set from the host's HA to the accelerator device (e.g., before the block (310) in the method (300).

[0060] There are at least two different options for moving modified data from the server to the accelerator device, which are exemplified by alternative sub-blocks (425, 430). In block (425), the HA receives only a flush command from the accelerator application. In one embodiment, this flush command is a flush to the target or a persistent flush to ensure that all modified data (e.g., modified data in CPU caches) has been updated in the target memory or persistent (i.e., non-volatile) memory of the accelerator device. In response to the flush command, the HA ensures that the modified data is flushed from the various caches of the host and copied to the local memory of the accelerator device.

[0061] The host CPUs invalidate the cache or maintain a cached clean copy of the contents corresponding to the data set for future low-latency re-reference. In this example, the HA may not receive an explicit invalidation command from the accelerator application, but the CPUs can proceed and invalidate a portion of their data set stored in their caches. In another example, while data is being homed by the accelerator device's SA, the CPUs can maintain their own local copies of the data set (which they can modify).

[0062] In an alternative embodiment, in block (430), HA receives flush and invalidate commands from the accelerator application. In response, the host CPUs flush and invalidate their caches.

[0063] In one embodiment, the flush and invalidation commands described in blocks (425, 430) are cache maintenance operations (CMOs) that automatically ensure that the modified data set (e.g., the most recent data set) is stored in local memory. In one embodiment, there are two types of CMOs that an accelerator application may issue: one that ensures only that the most recent data set is stored in local memory but allows the CPU to retain cached copies (e.g., only flush commands), and the other that ensures not only that the most recent data set is stored in local memory but also that no cached copies remain (e.g., flush and invalidation commands).

[0064] Returning to block (415), if the hardware of the accelerator device, rather than the software, is tasked with ensuring that the modified data set is received, the method proceeds to block (435) where the hardware waits until the accelerator application transfers ownership of the data set to the accelerator device (as discussed in block (310) of method (300). When this transfer of ownership occurs, method (400) proceeds to block (435) where the hardware of the accelerator device (e.g., RA) issues one or more CMOs to ensure that the modified data is transmitted to the accelerator device. As with block (420), there are at least two different options for moving the modified data from the host to the accelerator device, which are exemplified by alternative sub-blocks (445, 450). In block (445), the HA receives only a flush command from the RA. The actions that can be taken by the CPUs of the host may be the same as the actions described in block (425), so they are not repeated here.

[0065] Alternatively, in block (450), HA receives flush and invalidate commands from RA. In response, the host CPUs flush and invalidate their caches as described in block (430).

[0066] In one embodiment, the HA may have a final level cache. When ownership is transferred, the HA may ensure that the latest data set is not fixed in the final level cache but is instead copied to the local memory of the accelerator device. The final state of the final level cache is the same as the state of the CPU cache and corresponds to the actions of the CPU cache resulting from block (420) or block (440).

[0067] Some additional non-limiting advantages of the techniques described in Figures 3 and 4 are that, when memory is continuously homed by the host's HA (where data is not homed by the accelerator's SA), all caching and coherence actions by CPUs on memory are rapidly analyzed by the local HA, thereby maintaining high-performance CPU-memory access even when the collaboration framework exists in parallel. Additionally, the collaboration framework may request ownership of only a subset of accelerator-memory interactions to exercise separate or combined CMO and ownership requests to the HA. By transferring ownership of only the relevant subset, the accelerator gains a performance advantage as it does not need to claim ownership of the entire data set, as is required for the pure software coherence model of a conventional I / O model.

[0068] An additional advantage is that performance benefits can also be obtained in that a subset of CPU-memory interactions that do not overlap with accelerator-memory interactions is continued to be serviced by a local HA that maintains high-performance CPU-memory access. Additionally, in one embodiment, only a subset of data shared between the CPU and the accelerator is flushed from the CPU caches, while content not shared with the accelerator is continued to be cached in the CPUs, thereby providing the CPUs with low-latency access to that data. Furthermore, in the execution of a computational memory domain-specific architecture, accelerator-memory interactions occur on a low-latency, high-bandwidth dedicated path independent of CPU-memory interactions. A subset of software coherence in the collaboration framework can ensure that the metadata handoff between the host and the accelerator allows for CPU-memory execution and accelerator-memory execution independent of each other, thereby improving the overall cross-sectional bandwidth, latency, and program execution performance of both the CPU and accelerator devices.

[0069] Additionally, while the above embodiments specifically describe a hybrid hardware-software coherence framework for computational memory bottlenecks, similar techniques may be applied to related CPU-accelerator shared memory applications.

[0070] FIG. 5 illustrates an implementation of an FPGA (500) of an accelerator device (140), more specifically, an FPGA having a PL array comprising a plurality of different programmable tiles including transceivers (37), CLBs (33), BRAMs (34), input / output blocks ("IOBs") (36), configuration and clocking logic ("CONFIG / CLOCKS") (42), DSP blocks (35), special input / output blocks ("IO") (41) (e.g., configuration ports and clock ports), and other programmable logic (39), such as digital clock managers, analog-to-digital converters, system monitoring logic, etc. The FPGA may also include PCIe interfaces (40), analog-to-digital converters (ADCs) (38), etc.

[0071] In some FPGAs, each programmable tile may include at least one programmable interconnect element ("INT") (43) having connections to the input and output terminals (48) of a programmable logic element within the same tile, as illustrated by the examples included at the top of FIG. 5. Each programmable interconnect element (43) may also include connections for interconnecting segments (49) of adjacent programmable interconnect element(s) in the same tile or different tile(s). Each programmable interconnect element (43) may also include connections for interconnecting segments (50) of common routing resources between logic blocks (not shown). Common routing resources may include routing channels between logic blocks (not shown) containing tracks of interconnect segments (e.g., interconnect segments (50)) and switch blocks (not shown) for connecting the interconnect segments. The interconnection segments of general routing resources (e.g., interconnection segments (50)) may span one or more logic blocks. The programmable interconnection elements (43) taken together with the general routing resources implement a programmable interconnection structure ("programmable interconnection") for the illustrated FPGA.

[0072] In an exemplary implementation, the CLB (33) may include a configurable logic element ("CLE") (44) that can be programmed to implement user logic plus a single programmable interconnect element ("INT") (43). The BRAM (34) may include a BRAM logic element ("BRL") (45) in addition to one or more programmable interconnect elements. Typically, the number of interconnect elements included in a tile depends on the height of the tile. In the illustrated example, the BRAM tile has the same height as five CLBs, but other numbers (e.g., four) may also be used. The DSP block (35) may include a DSP logic element (DSPL) (46) in addition to an appropriate number of programmable interconnect elements. The IOB (36) may include, for example, two instances of an input / output logic element ("IOL") (47) in addition to one instance of a programmable interconnect element (43). As will be obvious to those skilled in the art, for example, the actual I / O pads connected to the I / O logic element (47) are typically not limited to the area of ​​the input / output logic element (47).

[0073] In the illustrated example, a horizontal zone near the center of the die (illustrated in FIG. 5) is used for configuration, clock, and other control logic. Vertical columns (51) extending from this horizontal zone or column are used to distribute clocks and configuration signals across the width of the FPGA.

[0074] Some FPGAs utilizing the architecture exemplified in FIG. 5 include additional logic blocks that disrupt the regular columnar structure that makes up much of the FPGA. The additional logic blocks may be programmable blocks and / or dedicated logic.

[0075] Note that FIG. 5 is intended to illustrate only an exemplary FPGA architecture. For example, the number of logic blocks within a row, the relative width of the rows, the number and order of the rows, the types of logic blocks included in the rows, the relative sizes of the logic blocks, and the interconnection / logic implementations included at the top of FIG. 5 are purely exemplary. For example, in an actual FPGA, wherever CLBs appear, more than one adjacent row of CLBs is typically included to facilitate the efficient implementation of user logic, but the number of adjacent CLB rows varies depending on the overall size of the FPGA.

[0076] Foregoing, reference is made to the embodiments presented in this disclosure. However, the scope of this disclosure is not limited to the specific described embodiments. Instead, any combination of the described features and elements is considered to embody and practice the embodiments being considered, whether or not they relate to different embodiments. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or prior art, whether a particular advantage is achieved by a given embodiment is not a limitation of the scope of this disclosure. Accordingly, the prior aspects, features, embodiments, and advantages are merely illustrative and are not considered as elements or limitations of the appended claims, except as explicitly stated in the claim(s).

[0077] As recognized by those skilled in the art, the embodiments disclosed herein may be implemented as systems, methods, or computer program products. Accordingly, aspects may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which may generally be referred to herein as “circuits, modules,” or “systems.” Furthermore, aspects may take the form of computer program products implemented on one or more computer-readable media(s) on which computer-readable program code is implemented.

[0078] Any combination of one or more computer-readable media(s) may be utilized. A computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any suitable combination of the above, but is not limited thereto. More specific examples of computer-readable storage media (a list that is not exhaustive) will include electrical connections having one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable CD-ROMs (compact disc read-only memory), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the context of the present invention, a computer-readable storage medium may be any type of medium capable of containing or storing a program for use by or in connection with an instruction execution system, device, or device.

[0079] A computer-readable signal medium may include a radio data signal in which computer-readable program code is implemented, for example, in the baseband or as part of a carrier wave. Such radio signals may take any of various forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium capable of communicating, propagating, or transmitting a program for use by or in connection with an instruction execution system, device, or device, and is not a computer-readable storage medium.

[0080] Program code implemented on a computer-readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wired, fiber optic cable, RF, etc., or any suitable combination thereof.

[0081] Computer program code for executing operations for aspects of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may be executed wholly on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN (local area network) or a WAN (wide area network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] Aspects of the present disclosure are described below with reference to flowchart examples and / or block diagrams of methods, apparatus (systems) and computer program products according to the embodiments presented in the present disclosure. It will be understood that each block of the flowchart examples and / or block diagrams, and combinations of blocks of the flowchart examples and / or block diagrams, may be implemented by computer program instructions. Such computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device for producing a machine, thereby creating means for instructions executed through the processor of the computer or other programmable data processing device to implement the functions / operations specified in the blocks of the flowchart and / or block diagrams.

[0083] These computer program instructions, which can command a computer, other programmable data processing device, or other device to function in a specific manner, may also be stored on a computer-readable medium, and the instructions stored on the computer-readable medium produce a manufactured article comprising instructions that implement a specific function / operation in a block or block of a flowchart and / or block diagram.

[0084] Computer program instructions can also be loaded into a computer, another programmable data processing device, or other devices so that a series of operation steps are performed on a computer, another programmable device, or other devices to create a computer-implemented process, thereby providing processes for implementing specific functions / operations in blocks or blocks of a flowchart and / or block diagram for instructions executed on a computer or other programmable device.

[0085] The flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block within the flowcharts or block diagrams may represent a module, segment, or part of instructions comprising one or more executable instructions for implementing a specific logic function(s). In some alternative implementations, the functions mentioned in the blocks may occur out of the order mentioned in the drawings. For example, depending on the accompanying functions, two consecutively drawn blocks may be executed substantially simultaneously, or the blocks may be executed in reverse order from time to time. It will also be noted that each block of the block diagrams and / or flowcharts and combinations of blocks of the block diagrams and / or flowcharts may be implemented by special-purpose hardware-based systems that perform specific functions or operations or perform combinations of special-purpose hardware and computer instructions.

[0086] Although the foregoing relates to specific examples, other and additional examples may be devised without departing from the basic scope of the present disclosure, and the scope of the present disclosure is determined by the subsequent claims.

Claims

Claim 1 A computing system comprising: a host including a processing unit, a home agent (HA), and an accelerator application; and an accelerator device communicably coupled to the host, wherein the accelerator device comprises a request agent (RA), a slave agent (SA), and a local memory configured to perform at least one accelerator function, wherein the local memory is part of the same coherent domain as the processing unit of the host, and wherein the accelerator application is configured to transfer ownership of the data set from the HA to the accelerator device in order to allow the RA to access the data set without first receiving permission from the HA, wherein the accelerator device is configured to store the latest copy of the data set in the local memory, service a memory request from the RA using the SA, and transfer ownership of the data set back to the HA, and wherein the accelerator application is configured to identify the data set, wherein the data set is a subportion of a memory block or memory page. Claim 2 delete Claim 3 A computing system according to claim 1, wherein, after transferring ownership of the data set from the HA to the accelerator device, the remainder of the memory block or the memory page continues to be homed by the HA. Claim 4 A computing system according to claim 1, wherein servicing a memory request from the RA using the SA is performed without receiving permission from the HA. Claim 5 A computing system according to claim 1, wherein the processing unit includes a cache, and the cache stores a modified version of a data set. Claim 6 A host including a processing unit, a home agent (HA), and an accelerator application; and includes an accelerator device communicably coupled to the host, wherein the accelerator device includes a request agent (RA) configured to perform at least one accelerator function, a slave agent (SA), and a local memory, wherein the local memory is part of the same coherent domain as the processing unit of the host, and the accelerator application is configured to transfer ownership of the data set from the HA to the accelerator device to allow the RA to access the data set without first receiving permission from the HA, the processing unit includes a cache, the cache stores a modified version of the data set, and the accelerator device sends a flush command to the HA instructing the processing unit to flush the cache so that the modified version of the data set is moved to the local memory after ownership of the data set is transferred to the accelerator device, stores the latest copy of the data set in the local memory, and uses the SA to receive a memory request from the RA A computing system configured to provide a service and transfer ownership of the data set back to the HA. Claim 7 A computing system according to claim 6, wherein the processing unit is configured to invalidate the modified version of the data set in the cache in response to the flush command; and to maintain a cached clean copy of the modified version of the data set for future low-latency re-reference. Claim 8 A host including a processing unit, a home agent (HA), and an accelerator application; and includes an accelerator device communicably coupled to the host, wherein the accelerator device includes a request agent (RA) configured to perform at least one accelerator function, a slave agent (SA), and local memory, wherein the local memory is part of the same coherent domain as the processing unit of the host, and the accelerator application is configured to transfer ownership of the data set from the HA to the accelerator device to allow the RA to access the data set without first receiving permission from the HA, the accelerator device is configured to store the latest copy of the data set in the local memory, service memory requests from the RA using the SA, and transfer ownership of the data set back to the HA, and the processing unit includes a cache, wherein the cache stores a modified version of the data set, and the accelerator application is configured to allow the processing unit to move the modified version of the data set to the local memory before ownership of the data set is transferred to the accelerator device. A computing system configured to send a flush command to the HA that commands the above cache to be flushed. Claim 9 In claim 8, the accelerator application is configured to send an invalidation command to the HA instructing the processing unit to invalidate the cache before ownership of the data set is transferred to the accelerator device, a computing system. Claim 10 An accelerator device comprising: a request agent (RA) including a compute engine configured to perform at least one accelerator function; a slave agent (SA) including a memory controller; and a memory, wherein the SA is configured to receive ownership of the data set from the HA of a host coupled to the accelerator device in order to allow the RA to access the data set without first receiving permission from the home agent (HA), and said ownership is transferred by a software accelerator application, and the memory controller is configured to service a request sent by the compute engine to access the data set stored in the memory when said ownership is transferred to the SA, and the RA is configured to transfer ownership back from the SA to the HA, and said data set is a sub-part of a memory block or memory page, and after the ownership of said data set is transferred from the HA to the SA, the remainder of said memory block or memory page continues to be homed by the HA. Claim 11 delete Claim 12 In paragraph 10, the accelerator device is configured to communicate with the host using a coherent interconnect protocol to extend the coherent domain of the host to include the memory and the SA in the accelerator device. Claim 13 A method comprising: identifying a data set using an accelerator application running on a host — said data set is a sub-part of a memory block or memory page, and after transferring ownership of said data set from the home agent (HA) of said host to the accelerator device, the remainder of said memory block or said memory page continues to be homed by said HA —; transferring ownership of said data set from said HA of said host to said accelerator device by the accelerator application running on said host to allow said accelerator device to access said data set without first receiving permission from said HA — said accelerator device is communicably coupled to said host and the coherent domain of said host extends to said accelerator device —; moving said data set to the local memory of said accelerator device; servicing a memory request from a request agent (RA) of said accelerator device using a slave agent (SA) of said accelerator device; and transferring ownership back from said accelerator device to said HA. Claim 14 delete Claim 15 A method comprising: transferring ownership of a data set from the HA of the host to the accelerator device in order to allow the accelerator device to access the data set without first receiving permission from the home agent (HA) by an accelerator application running on the host—the accelerator device is communicably coupled to the host and the coherent domain of the host extends to the accelerator device—; after transferring ownership to the accelerator device, invalidating the modified data set in the cache in response to a cache maintenance operation (CMO) issued by the accelerator device; moving the data set to the local memory of the accelerator device; servicing a memory request from a request agent (RA) of the accelerator device using a slave agent (SA) of the accelerator device; and transferring ownership back from the accelerator device to the HA.

Images