Machine learning model updates to ML accelerators

Abstract

Examples herein describe a peripheral I / O device having a hybrid gateway that allows the device to have an I / O domain and a coherence domain. Thus, computing resources in the coherence domain of the peripheral I / O device may communicate with the host in a manner similar to CPU-to-CPU communications in the host. Dual domains in a peripheral I / O device may be used for machine learning (ML) applications. Although I / O devices may be used as ML accelerators, these accelerators previously use only I / O domains. In embodiments herein, computing resources may be partitioned between an I / O domain and a coherence domain, with an ML engine in the I / O domain and an ML model in the coherence domain. An advantage doing so is that the ML model can be coherently updated using a reference ML model stored in the host.

Description

[0001] Description of the case

[0002] This application is a divisional application of the Chinese invention patent application with application number 202080031252.4 and name “Updating machine learning models for ML accelerators” filed on April 10, 2020. Technical Field

[0003] Examples of the present disclosure generally relate to executing a machine learning model in a peripheral I / O device that supports an I / O domain and a coherent domain. Background Art

[0004] In the traditional I / O model, the host computing system interfaces with its peripheral I / O devices while using a custom I / O device driver unique to the peripheral I / O device to perform accelerator tasks or functions. Having multiple I / O devices or even multiple instances of the same I / O device means that the host interfaces with multiple I / O device drivers or multiple running copies of the same I / O device driver. This can lead to security and reliability issues because the I / O device driver is usually developed by the vendor that provides the peripheral I / O device, but must be integrated with all software and hardware in the host computing system.

[0005] At the same time, the hardware cache coherent shared memory multiprocessor paradigm utilizes a common, instruction set architecture (ISA)-independent docking model when executing tasks or functions on a multiprocessor CPU. The common, ISA-independent (e.g., C code) docking model scales with the number of processing units and the amount of shared memory available to those processing units. Traditionally, peripheral I / O devices have not been able to benefit from the coherent paradigm used by the CPU executing on the host computing system. Summary of the invention

[0006] Techniques for executing machine learning models using I / O domains and coherence domains in a peripheral device are described. One example is a peripheral I / O device that includes: a hybrid gateway configured to communicatively couple the peripheral I / O device to a host; I / O logic including a machine learning (ML) engine assigned to the I / O domain; and coherence logic including an ML model assigned to the coherence domain, wherein the ML model shares the coherence domain with computing resources in the host.

[0007] In some embodiments, the hybrid gateway is configured to use a coherent interconnect protocol to extend the coherence domain of the host into the peripheral I / O devices.

[0008] In some embodiments, the hybrid gateway includes an update agent configured to update the ML model using a cache coherent shared memory multiprocessor paradigm in response to changes to a reference ML model stored in a memory associated with the host.

[0009] In some embodiments, updating the ML model using a cache coherent shared memory multiprocessor paradigm results in only a portion of the ML model being updated when a reference ML model is updated.

[0010] In some embodiments, the peripheral I / O device includes a network on chip (NoC) coupled to the I / O logic and the coherent logic, wherein at least one of the NoC, programmable logic (PL) to PL messaging, and wired signaling is configured to allow parameters associated with a layer in the ML model to be passed from the coherent logic to the I / O logic.

[0011] In some embodiments, the ML engine is configured to process the ML dataset received from the host using parameters in the ML model.

[0012] In some embodiments, a peripheral I / O device includes: a PL array, wherein a first plurality of PL blocks in the PL array are part of an I / O logic and are assigned to an I / O domain, and a second plurality of PL blocks in the PL array are part of a coherent logic and are assigned to a coherent domain; and a plurality of memory blocks, wherein a first subset of the plurality of memory blocks are part of the I / O logic and are assigned to the I / O domain, and a second subset of the plurality of memory blocks are part of the coherent logic and are assigned to the coherent domain. In addition, the first subset of the plurality of memory blocks can communicate with the first plurality of PL blocks but not directly with the second plurality of PL blocks, and the second subset of the plurality of memory blocks can communicate with the second plurality of PL blocks but not directly with the first plurality of PL blocks.

[0013] An example described herein is a computing system including a host and a peripheral I / O device. The host includes a memory storing a reference ML model and a plurality of CPUs forming a coherent domain together with the memory. The I / O device includes an I / O logic and a coherent logic, the I / O logic including an ML engine assigned to the I / O domain, and the coherent logic including an ML model assigned to the coherent domain and the memory and the plurality of CPUs in the host.

[0014] In some embodiments, wherein the peripheral I / O device is configured to use a coherent interconnect protocol to extend the coherence domain into the peripheral I / O device.

[0015] In some embodiments, the peripheral I / O device includes an update agent configured to update the ML model using a cache coherent shared memory multiprocessor paradigm in response to changes to a reference ML model stored in the memory.

[0016] In some embodiments, updating the ML model using a cache coherent shared memory multiprocessor paradigm results in only a portion of the ML model being updated when a reference ML model is updated.

[0017] In some embodiments, the host includes a plurality of reference ML models stored in a memory, and the peripheral I / O device includes a plurality of ML models corresponding to the plurality of reference ML models in a coherent domain, and a plurality of ML engines assigned to the I / O domain, wherein the plurality of ML engines are configured to execute independently of each other.

[0018] In some embodiments, a peripheral I / O device includes: a PL array, wherein a first plurality of PL blocks in the PL array are part of an I / O logic and are assigned to an I / O domain, and a second plurality of PL blocks in the PL array are part of a coherent logic and are assigned to a coherent domain; and a plurality of memory blocks, wherein a first subset of the plurality of memory blocks are part of the I / O logic and are assigned to the I / O domain, and a second subset of the plurality of memory blocks are part of the coherent logic and are assigned to the coherent domain. In addition, the first subset of the plurality of memory blocks can communicate with the first plurality of PL blocks but not directly with the second plurality of PL blocks, and the second subset of the plurality of memory blocks can communicate with the second plurality of PL blocks but not directly with the first plurality of PL blocks.

[0019] One example described herein is a method that includes: updating a sub-portion of a reference ML model in a memory associated with a host; updating a subset of a cached ML model in coherent logic associated with a peripheral I / O device coupled to the host, wherein the memory of the host and the coherent logic of the peripheral I / O device are in the same coherent domain; retrieving the updated subset of the cached ML model from the coherent domain; and processing an ML data set according to parameters in the retrieved subset of the cached ML model using an ML engine, wherein the ML engine is located in I / O logic in the peripheral I / O device assigned to the I / O domain.

[0020] In some embodiments, retrieving the updated subset of the cached ML model from the coherence domain is performed using at least one of NoC, PL to PL messaging, and wired signaling that communicatively couples the coherence logic assigned to the coherence domain to the I / O logic assigned to the I / O domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to be able to understand the manner of the above-mentioned features in detail, a more specific description may be made by reference to example implementations, which are briefly summarized above, some of which are illustrated in the accompanying drawings. However, it should be noted that the accompanying drawings only illustrate typical example implementations and therefore should not be considered as limiting the scope thereof.

[0022] Figure 1 is a block diagram of a host coupled to a peripheral I / O device having an I / O domain and a coherence domain according to one example.

[0023] Figure 2 is a block diagram of a peripheral I / O device with programmable logic, memory, and on-chip network according to one example, the peripheral I / O device being logically divided into an I / O domain and a coherence domain.

[0024] Figure 3 is a block diagram of a peripheral I / O device with a machine learning model and a machine learning engine according to an example.

[0025] Figure 4 is a flowchart for updating a machine learning model in a coherent domain of an I / O device according to an example.

[0026] Figure 5 is a block diagram of an I / O expansion box including multiple I / O devices according to one example.

[0027] Figure 6 is a flowchart for updating a machine learning model cached in multiple I / O devices according to an example.

[0028] Figure 7 is a flowchart of updating a machine learning model using a recursive learning algorithm according to an example.

[0029] Figure 8 A field programmable gate array implementation of a programmable IC according to one example is illustrated. DETAILED DESCRIPTION

[0030] Various features will be described below with reference to the accompanying drawings. It should be noted that the drawings may or may not be drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. It should be noted that these figures are only for the convenience of describing the features. They are not intended to be an exhaustive description of the specification or as a limitation on the scope of the claims. Additionally, even if not so stated, or if not so explicitly described, the illustrated examples do not need to have all the aspects or advantages shown. The aspects or advantages described in conjunction with a particular example are not necessarily limited to that example and may be practiced in any other example.

[0031] The examples herein describe peripheral I / O devices with hybrid gateways that allow devices to have I / O domains and coherent domains. That is, I / O devices can enjoy the benefits of a traditional I / O model in which an I / O device driver manages some computing resources in an I / O device, and the benefits of adding other computing resources in an I / O device to the same coherent domain used by a processor (e.g., a central processing unit (CPU)) in a host computing system. Therefore, computing resources in a coherent domain of a peripheral I / O device can communicate with the host in a manner similar to CPU-to-CPU communication in a host. This means that computing resources can utilize coherent class features, such as direct communication, more efficient memory usage, non-uniform memory access (NUMA) awareness, etc. At the same time, computing resources in an I / O domain can benefit from the advantages of a traditional I / O device model that provides efficiency when large memory transfers (e.g., direct memory access (DMA)) are performed between a host and an I / O device.

[0032] Dual domains in peripheral I / O devices can be used for machine learning (ML) applications. Although I / O devices can be used as ML accelerators, these accelerators previously only used I / O domains. In an embodiment of the present invention, computing resources can be divided between I / O domains and coherent domains, where ML engines are assigned to I / O domains and ML models are stored in coherent domains. The advantage of this is that ML models can be updated coherently using reference ML models stored in the host. That is, several types of ML applications benefit from being able to quickly (e.g., in real time or with low latency) update one or more ML models in I / O devices. Storing ML models in coherent domains (rather than I / O domains) means that a cache coherent shared memory multiprocessor paradigm can be used to update ML models, which is much faster than relying on traditional I / O domain models (e.g., direct memory access (DMA)). However, the ML engine can be executed in the I / O domain of the peripheral I / O device. This is beneficial because ML engines often process large amounts of ML data, which are more efficiently transferred between I / O devices and hosts using DMA rather than a cache coherent paradigm.

[0033] Figure 1 is a block diagram of a host 105 coupled to a peripheral I / O device 135 having an I / O domain and a coherence domain according to one example. Figure 1The computing system 100 in includes a host 105, which is communicatively coupled to a peripheral I / O device 135 using a PCIe connection 130. The host 105 can represent a single computer (e.g., a server) or multiple interconnected physical computing systems. In any case, the host 105 includes an operating system 110, multiple CPUs 115, and a memory 120. The OS 110 can be any OS capable of performing the functions described herein. In one embodiment, the OS 110 (or a hypervisor or a kernel) establishes a cache-coherent shared memory multiprocessor paradigm for the CPU 115 and the memory 120. In one embodiment, the CPU 115 and the memory 120 are managed by the OS (or managed by the kernel / hypervisor) to form a coherent domain that follows the cache-coherent shared memory multiprocessor paradigm. However, as described above, the traditional I / O model means that the peripheral I / O device 135 (and all its computing resources 150) are excluded from the coherent domain established in the host 105. Instead, the host 105 relies on I / O device drivers 125 stored in its memory 120, which manage computing resources 150 in the I / O devices 135. That is, the peripheral I / O devices 135 are controlled by the I / O device drivers 125 and can be accessed through the I / O device drivers 125.

[0034] In embodiments herein, the shared memory multiprocessor paradigm, along with all of the performance advantages, software flexibility, and reduced overhead of that paradigm, is available to peripheral I / O devices 135. Furthermore, adding computing resources in I / O devices 135 to the same coherent domain as CPU 115 and memory 120 allows for a common, ISA-independent development environment. Figure 1 As shown, some computing resources 150 in peripheral I / O device 135 are assigned to coherence domain 160 , which is the same coherence domain 160 used by computing resources (eg, CPU 115 and memory 120 ) in host 105 .

[0035] While computing resources 150C and 150D are logically assigned to coherence domain 160, computing resources 150A and 150B are assigned to I / O domain 145. Thus, I / O device 135 benefits from the assignment of computing resources 150 to both domains 145, 160. I / O domain 145 provides efficiency when transferring large memories between host 105 and I / O device 135, while coherence domain 160 provides the performance advantages, software flexibility, and reduced overhead described above. By logically partitioning hardware computing resources 150 (e.g., programmable logic, network on chip (NoC), data processing engine, and / or memory) into I / O domain 145 and coherence domain 160, I / O device 135 can benefit from both types of paradigms.

[0036] To enable the host 105 to send and receive I / O data traffic and coherent data traffic, the peripheral I / O device 135 includes a hybrid gateway 140 that separates data received on the PCIe connection 130 into I / O data traffic and coherent data traffic. The I / O data traffic is forwarded to computing resources 150A and 150B in the I / O domain 145, while the coherent data traffic is forwarded to computing resources 150C and 150D in the coherent domain 160. In one embodiment, the hybrid gateway 140 can process the I / O and coherent data traffic in parallel, so that the computing resources 150 in the I / O domain 145 can execute in parallel with the computing resources 150 in the coherent domain 160. That is, the host 105 can assign tasks to computing resources 150 in both the I / O domain 145 and the coherent domain 160 that can execute those tasks in parallel.

[0037] The peripheral I / O device 135 can be many different types of I / O devices, such as a pluggable card (plugged into an expansion slot in the host 105 or a separate expansion box), a system on a chip (SoC), a graphics processing unit (GPU), a field programmable gate array (FPGA), etc. Therefore, although many embodiments discuss I / O devices 135 that include programmable logic (e.g., a programmable logic array), these embodiments can be applied to I / O devices 135 that do not have programmable logic but only contain hardened circuits (which can be software programmable). In addition, although the embodiments herein discuss dividing the computing resources 150 into two domains, in other embodiments, the hybrid gateway 140 can be modified to support additional domains or multiple sub-domains within the I / O domain and the coherent domain 145, 160.

[0038] In one embodiment, the hybrid gateway 140 and the host 105 use a coherent interconnect protocol to extend the coherence domain 160 into the peripheral I / O devices 135. For example, the hybrid gateway 140 may use the Cache Coherent Interconnect for Accelerators (CCIX) for extending the coherence domain 160 within the device 135. CCIX is a high-performance, chip-to-chip interconnect architecture that provides a cache coherence framework for heterogeneous system architectures. CCIX brings kernel-managed semantics to the peripheral devices 135. Cache coherence is always automatically maintained between the CPU on the host 105 and the various other accelerators in the system that may be located on any number of peripheral I / O devices.

[0039] However, in addition to CCIX, other coherent interconnect protocols such as Quick Path Interconnect (QPI), Omni-Path, Infinity Fabric, NVLink, or OpenCAPI may be used to extend the coherent domain in the host 105 to include computing resources 135 in peripheral I / O devices. That is, the hybrid gateway may be customized to support any type of coherent interconnect protocol that facilitates forming a coherent domain including computing resources in the I / O devices 135.

[0040] Figure 2 2 is a block diagram of a peripheral I / O device 135 having a programmable logic (PL) array 205, a memory block 220, and a NoC 230, the peripheral I / O device 135 being logically divided into an I / O domain 145 and a coherent domain 160 according to one example. In this example, the PL array 205 is formed of a plurality of PL blocks 210. The blocks may be individually assigned to an I / O domain 145 or a coherent domain 160. That is, PL blocks 210A and 210B are assigned to the I / O domain 145, while PL blocks 210C and 210D are assigned to the coherent domain 160. In one embodiment, a set of PL blocks 210 assigned to an I / O domain is mutually exclusive with a set of PL blocks 210 assigned to a coherent domain, such that there is no overlap between the blocks (e.g., no PL block 210 is assigned to both an I / O domain and a coherent domain).

[0041] In one embodiment, assigning a hardware resource to an I / O domain 145 or a coherence domain 160 does not affect (or dictate) the physical location of the hardware resource in I / O device 135. For example, PL blocks 210A and 210C may be assigned to different domains even though these blocks are adjacent to each other in PL array 205. Thus, while the physical location of the hardware resources may be considered when logically assigning hardware resources in I / O device 135 to I / O domains 145 and coherence domains 160, this is not necessary.

[0042] The I / O device 135 also includes memory controllers 215 assigned to the I / O domain 145 and the coherence domain 160. In one embodiment, due to the physical interconnection between the memory controllers 215 and the corresponding memory blocks 220, assigning one of the memory controllers 215 to the I / O domain 145 or the coherence domain 160 means that all memory blocks 220 connected to the memory controller 215 are also assigned to the same domain. For example, the memory controller 215 can be coupled to a fixed set of memory blocks 220 (which are not coupled to any other memory controller 215). Therefore, the memory blocks 220 can be assigned to the same domain as the memory controller 215 to which they are coupled. However, in other embodiments, the memory blocks 220 coupled to the same memory controller 215 can be assigned to different domains.

[0043] In one embodiment, the NoC includes an interface element that allows hardware elements in the I / O device 135 (e.g., configurable data processing engines, memory blocks 220, PL blocks 210, etc.) to use the NoC 230 to send and receive data. In one embodiment, rather than using programmable logic to form the NoC 230, some or all of the components that form the NoC are hardened. In any case, the NoC 230 can be logically divided between the I / O domain 145 and the coherence domain 160. In one embodiment, rather than allocating different portions of the NoC 230 to the two domains, the parameters of the NoC are configured to provide different service levels to data traffic corresponding to the I / O domain 145 and the coherence domain 160. That is, data traffic for two domains flowing in the NoC 230 can use the same hardware elements (e.g., switches and communication links), but is processed differently by the hardware elements. For example, the NoC 230 can provide different quality of service (QoS), latency, and bandwidth for two different domains. In addition, the NoC 230 may also isolate the traffic of the I / O domain 145 from the traffic of the coherence domain 160 for security reasons.

[0044] In another embodiment, the NoC 230 may prevent computing resources in an I / O domain 145 from communicating with computing resources in a coherent domain 160. However, in one embodiment, it may be advantageous to allow computing resources assigned to an I / O domain 145 to communicate with computing resources assigned to a coherent domain 160. Previously, this communication would occur between the I / O device driver 125 and the OS in the host 105. Instead, inter-domain communication may occur within an I / O device 135 using the NoC 230 (if the computing resources are far apart in a device 135) or fabric-to-fabric connections in the PL array 205 (if two PL blocks 210 assigned to two different domains are close together and need to communicate).

[0045] Figure 3 is a block diagram of a peripheral I / O device 135 having an ML model 345 and an ML engine 335 according to one example. Figure 3In the embodiment, the host 105 is coupled to the host-attached memory 305 storing ML data and results 310 and the reference ML model 315. The ML data and results 310 include data sent by the host 105 to the peripheral I / O device 135 (e.g., ML accelerator) for processing, and the results received back by the host 105 from the I / O device 135. On the other hand, the reference ML model 315 defines the layers and parameters of the ML algorithm, which the peripheral I / O device 135 uses to process the ML data. The reference ML model 315 may also include multiple ML models, each of which defines the layers and parameters of multiple ML algorithms for processing ML data, so that the host receives results across ML algorithms. The embodiments herein are not limited to a specific ML model 315 and may include binary classification, multi-class classification, regression, neural network (e.g., convolutional neural network (CNN) or recurrent neural network (RNN)), etc. The ML model 315 may define the number of layers, how the layers are interconnected, the weights of each layer, etc. Furthermore, while host-attached storage 305 is shown as being separate from host 105 , in other embodiments, ML data and results 310 and ML model 315 are stored in storage within host 105 .

[0046] The host 105 may update the reference ML model 315. For example, as more data becomes available, the host 105 may change some weights in a particular layer of the reference ML model 315, change how the layers are interconnected, or add / remove layers in the ML model 315. As described below, these updates in the reference ML model 315 may be mirrored in the ML model 345 stored (or cached) in the peripheral I / O device 135.

[0047] The hybrid gateway 140 allows the coherence domain of the host 105 to be extended to include hardware elements in the peripheral I / O devices 135. Additionally, the hybrid gateway 140 establishes an I / O domain that can use a traditional I / O model, where the hardware resources assigned to the domain are managed by I / O device drivers. To this end, the hybrid gateway includes an I / O and DMA engine 320 and an update agent 325, the I / O and DMA engine 320 passing I / O domain traffic between the hardware in the host 105 and the peripheral I / O devices 135 assigned to the I / O domain, and the update agent 325 passing coherence domain traffic between the hardware in the host 105 and the peripheral I / O devices 135 assigned to the coherence domain.

[0048] In this example, hybrid gateway 140 (as well as I / O and DMA engine 320 and update agent 325) is connected to NoC 230, which facilitates communication between gateway 140 and I / O logic 330 and coherence logic 340. I / O logic 330 represents hardware elements in peripheral I / O devices 135 that are assigned to I / O domains, while coherence logic 340 represents hardware elements assigned to coherence domains. In one embodiment, I / O logic 300 and coherence logic 340 include Figure 2 2 and 3. The PL block 210 and memory block 220 are shown. That is, a portion of the PL block 210 and memory block 220 form the I / O logic 330, while another portion forms the coherent logic 340. However, in another embodiment, the I / O logic 300 and the coherent logic 340 may not include any PL but include hardened circuitry (which may be software programmable). For example, the peripheral I / O device 135 may be an ASIC or a dedicated processor that does not include a PL.

[0049] As shown, ML engine 335 is executed using I / O logic 330, while ML model 345 is stored in coherence logic 340. Thus, ML model 345 is in the same coherence domain as host-attached memory 305 and a CPU (not shown) in host 105. In contrast, ML engine 335 is not part of the coherence domain, and therefore, is not coherently updated when data stored in memory 305 is updated or otherwise changed.

[0050] Additionally, the peripheral I / O device 135 is coupled to an attached memory 350 that stores the ML model 345 (which may be a cached version of the ML model 345 stored in the coherence logic 340). For example, the peripheral I / O device 135 may not store the entire ML model 345 in the coherence logic 340. Instead, the entire ML model 345 may be stored in the attached memory 350, while certain portions of the ML model 345 currently used by the ML engine 335 are stored in the coherence logic 340. In any case, the memory elements in the attached memory 350 storing the ML model 345 are part of the same coherence domain as the coherence logic 340 and the host 105.

[0051] In contrast, ML data set 355 is stored in a memory element assigned to the I / O domain. For example, ML engine 335 may retrieve data stored in ML data set 355, process the data according to ML model 345, and then store the processed data back into attached memory 350. Thus, in this manner, ML engine 335 and ML data set 355 are assigned to hardware elements in the I / O domain, while ML model 345 is assigned to hardware elements in the coherence domain.

[0052] Although Figure 3One ML engine and one ML model are illustrated, but the peripheral I / O device 135 can execute any number of ML engines and ML models. For example, unless the image includes both object A and object B, the first ML model may be good at recognizing object A in the captured image in most cases. However, the second ML model cannot recognize object A in many cases, but is good at distinguishing object A from object B. Therefore, the system administrator can instruct the ML engine 335 to execute two different ML models (e.g., the two ML models stored in the coherence logic 340). In addition, executing the ML engine 335 and the ML model 345 may only require a portion of the available computing resources in the peripheral I / O device 135. In this case, the administrator can execute another ML engine in the device 135 using its corresponding ML model. In other words, the I / O logic 330 can execute two ML engines, while the coherence logic 340 stores two ML models. These ML engine / model pairs can be executed independently of each other.

[0053] In addition, the allocation of computing resources to the I / O domain and the related domain can be dynamic. For example, the system administrator can determine that there are not enough resources in the I / O domain for the ML engine 335, and reconfigure the peripheral I / O devices 135 so that the computing resources previously allocated to the related domain are now allocated to the I / O domain. For example, the PL and memory blocks previously allocated to the coherent logic 340 can be reallocated to the I / O logic 330 - for example, the administrator may want to execute two ML engines or require the ML engine 335 to execute two ML models. The I / O devices 135 can be reconfigured with the new allocation, and the hybrid gateway 140 can support the operation of the I / O domain and the coherent domain at the same time.

[0054] Figure 4 4 is a flow chart of a method 400 for updating an ML model in a coherent domain of an I / O device according to an example. At box 405, the host updates a portion of a reference ML model in its memory. For example, the OS in the host (or a software application in the host) can execute a training algorithm to change or adjust the reference model. In one embodiment, the ML model is used to evaluate an image to detect a specific object. When an object is detected by the ML engine, the host can rerun the training algorithm, which results in an update to the ML model. That is, because detecting an object in an image can improve the training data, the host can decide to rerun the training algorithm (or a portion of the training algorithm) that can adjust the reference ML model. For example, the host can change the weights corresponding to one or more layers in the reference ML model, or change the way the layers are interconnected. In another example, the host can add or delete layers in the reference ML model.

[0055] In one embodiment, the host updates only a portion of the reference ML model. For example, when the host changes the weights corresponding to one or more layers, the remaining layers in the reference ML model remain unchanged. Therefore, after rerunning the training algorithm, most of the data defining the ML model can remain unchanged. For example, the reference ML model may have a total of 20MB of data, but the update may only affect 10% of that data. Under the traditional I / O device paradigm, an update to the reference ML model, no matter how small, requires the host to transfer the entire ML model (updated data and non-updated data) to the peripheral I / O device. However, by storing the ML model in a coherent domain of the peripheral I / O device, it is possible to avoid transferring the entire reference ML model to the I / O device every time there is an update.

[0056] At block 410, the host updates only a subset of the cached ML models for the peripheral I / O device. More specifically, the host transmits the updated data in the reference ML model to the peripheral device at block 410. The transfer occurs within the coherent domain and can therefore behave like a transfer between memory elements within the CPU-memory complex of the host. This is particularly useful in ML or artificial intelligence (AI) systems that rely on frequent (or low latency) updates to ML models in ML accelerators (e.g., peripheral I / O devices).

[0057] In another example, when the same ML model is distributed across many different peripheral I / O devices, it may be beneficial to place the ML model in a coherent domain of the I / O devices. That is, a host may be attached to multiple peripheral I / O devices with the same ML model. Thus, at each of the peripheral I / O devices, the coherent domain may be used to update only the data that has changed in the reference ML model, rather than updating the entire reference ML model.

[0058] At block 415, the ML engine retrieves the updated portion of the ML model in the peripheral I / O device from the coherence domain. For example, while the NoC may be able to separate I / O domain traffic and coherence domain traffic, the NoC can facilitate communication between hardware elements assigned to the I / O domain and the coherence domain when needed. But the NoC is only one of the transport mechanisms that can facilitate communication between the coherence domain and the I / O domain. Other examples include direct PL to PL messaging or wired signaling, and communication via metadata written to a shared memory buffer between the two domains. Thus, the peripheral I / O device can transfer data from the ML model to the ML engine. Doing so enables the ML engine to process the ML data set according to the ML model.

[0059] In one embodiment, the ML engine may retrieve only a portion of the ML model at any particular time. For example, the ML engine may retrieve parameters (e.g., weights) for one layer and configure the I / O logic to execute that layer in the ML model. Once completed, the ML engine may retrieve parameters for the next layer of the ML model, and so on.

[0060] At block 420, the I / O logic in the peripheral I / O device processes the ML data set using the ML engine in the I / O domain according to the parameters in the ML model. The ML engine may receive the ML data set from the host using I / O domain technology such as DMA. The ML data set may be stored in the peripheral I / O device or in an attached memory.

[0061] The ML engine returns the results of processing the ML data set using the parameters in the ML model to the host at block 425. For example, once completed, a DMA engine in the hybrid gateway may initiate a DMA write to transfer the processed data from the peripheral I / O device (or attached memory) to the host using an I / O device driver.

[0062] Figure 5 is a block diagram of an I / O expansion box 500 including a plurality of I / O devices 135 according to one example. Figure 5 In the embodiment, the host 105 communicates with multiple peripheral I / O devices 135, which may be separate ML accelerators (e.g., separate accelerator cards). In one embodiment, the host 105 may assign different tasks to different peripheral I / O devices 135. For example, the host 105 may send a different ML data set to each of the peripheral I / O devices 135 for processing.

[0063] In this embodiment, the same ML model 525 is executed on all peripheral I / O devices 135. That is, the reference ML model 315 in the host 105 is provided to each of the I / O devices 135 so that these devices 135 use the same ML model 525. As an example, the host 105 may receive feeds from multiple cameras (e.g., multiple cameras for autonomous vehicles or multiple cameras in an urban area). In order to process the data generated by the cameras in a timely manner, the host 105 may chunk the data and send different feeds to different peripheral I / O devices 135 so that these devices 135 can use the same ML model 525 to evaluate the data sets in parallel. Therefore, in an ML or AI environment where fast response time is important or desired, using an I / O expansion box 500 with multiple peripheral I / O devices 135 may be preferred.

[0064] In addition to storing the ML model 525 in the peripheral I / O device 135, the expansion box 500 includes a coherent switch 505 that is separate from the I / O device 135. Nevertheless, the coherent switch 505 is also in the same coherent domain as the hardware resources in the host 105 and the cache 520 in the peripheral I / O device 135. In one embodiment, according to the NUMA arrangement, the cache 510 in the coherent switch 505 is another cache layer located between the cache 520 in the peripheral I / O device 135 and the memory element storing the reference ML model 315.

[0065] Although the host 105 may transmit N copies of the reference ML model 315 (where N is the total number of peripheral I / O devices 135 in the container) to each device 135 when a portion of the reference ML model 315 is updated, because the caches 520 and 510 are in the same coherent domain, only the updated portion of the reference ML model 315 is transmitted to the caches 510 and 520. Therefore, Figure 5 The arrangement in can scale better than the embodiment in which the ML model 525 is stored in the hardware resources of the I / O domain allocated to the peripheral I / O device 135.

[0066] Figure 6 is a flow chart of a method 600 for updating a machine learning model cached in multiple I / O devices according to an example. In one embodiment, the method 600 is used to update multiple copies of the ML model, such as Figure 5 In the example shown in , these copies are stored in multiple peripheral I / O devices coupled to the host. At block 605, the host updates the reference ML model stored in the host memory. The reference ML model can be stored in local memory or attached memory. In either case, the reference ML model is part of a coherent domain shared by, for example, a CPU in the host.

[0067] At block 610, method 600 branches depending on whether a push model or a pull model is used to update the ML model. If the pull model is used, method 600 proceeds to block 615, where the host invalidates a subset of cached ML models in switches and peripheral I / O devices. Figure 5 , the host 105 invalidates the ML model 515 stored in the cache 510 in the switch 505 and the ML model 525 stored in the cache 520 in the peripheral I / O device 135. Because the ML model 525 is in the same coherent domain as the host 105, the host 105 does not need to invalidate all of the data of the ML model 525, but only the subset that has been changed in response to updating the reference ML model 315.

[0068] At block 620, the update agent in the peripheral I / O device retrieves the updated portion of the reference ML model from the host memory. In one embodiment, block 620 is performed in response to the ML engine (or any other software or hardware actor in the coherent switch or peripheral I / O device) attempting to access an invalid subset of the ML model. That is, if the ML engine attempts to retrieve data from an ML model that has not been invalidated in the cache, the requested data is provided to the ML engine. However, if the ML engine attempts to retrieve data from an invalid portion of the cache (also known as a cache miss), doing so triggers block 620.

[0069] In one embodiment, after determining that the requested data has been invalidated at the local cache in the peripheral I / O device (e.g., cache 520), the update agent first attempts to determine whether the requested data is available in the cache of the coherent switch (e.g., cache 510). However, as part of executing block 615, the host invalidates the same subset of the caches in the coherent switch and the peripheral I / O device 135. Doing so causes the update agent to retrieve update data from the reference ML model stored in the host.

[0070] In the pull model, after there is a cache miss (e.g., when the ML engine requests an invalidated cache entry from the ML model), the updated data in the reference ML model is retrieved. Thus, the peripheral I / O device may perform block 620 at different times (e.g., as needed) depending on when the ML engine (or any other actor in the device) requests the invalidated portion of the ML model.

[0071] In contrast, if the ML model is updated using a push model, at block 610, method 600 proceeds to block 625, where the host pushes the updated portion to the caches in the switches and peripheral I / O devices. In this model, the host controls when the cached ML models in the peripheral I / O devices are updated, rather than updating those ML models when there is a cache miss. The host can push the update data to the peripheral I / O devices in parallel or sequentially. In any case, the host does not have to push out all the data in the reference ML model, but only the portion of the reference ML model that is updated or changed.

[0072] Figure 7 700 is a flow chart of a method 700 for updating a machine learning model using a recursive learning algorithm according to an example. In one embodiment, the method 700 can be used to update a reference ML model using information obtained from executing the ML model in a peripheral I / O device. At block 705, the peripheral I / O device (or host) identifies false positives in the result data generated by the ML engine when executing the ML model. For example, the ML model may be designed to recognize a specific object or person in an image, but occasionally provides a false positive (e.g., identifies an object or person, but the object or person is not actually in the image).

[0073] At block 710, the host updates a reference ML model in the host using a recursive learning algorithm. In one embodiment, the recursive learning algorithm updates the training data used to train the reference ML model. In response to a false positive, the host can update the training data and then rerun at least a portion of the training algorithm using the updated training data. Thus, the recursive learning algorithm can update the reference ML model in real time using the result data provided by the ML engine.

[0074] At block 715, the host updates the cached ML model(s) using the coherent domain. For example, the host can update one or more ML models in the peripheral I / O devices using the pull model described in blocks 615 and 620 or the push model described in block 625 of method 600. Thus, by identifying false positives in result data generated by one or more of the peripheral I / O devices (e.g., one of the ML accelerators), the host can update the reference ML model. The host can then use the push model or the pull model to update the cached ML models on all peripheral I / O devices coupled to the host.

[0075] Figure 8 800 implementation of an I / O peripheral 135 is illustrated, and more specifically uses Figure 2 205 in the FPGA, the implementation includes a large number of different programmable tiles, including transceivers 37, CLBs 33, BRAMs 34, input / output blocks ("IOBs") 36, configuration and clock logic ("CONFIG / CLOCKS") 42, DSP blocks 35, dedicated input / output blocks ("IO") 41 (e.g., configuration ports and clock ports), and other programmable logic 39, such as digital clock managers, analog-to-digital converters, system monitoring logic, etc. The FPGA may also include a PCIe interface 40, an analog-to-digital converter (ADC) 38, etc.

[0076] In some FPGAs, each programmable tile may include at least one programmable interconnect element ("INT") 43, such as Figure 8As shown in the example included at the top, the programmable interconnect element has connections to the input and output terminals 48 of the programmable logic elements within the same tile. Each programmable interconnect element 43 may also include connections to the interconnect segments 49 of the adjacent programmable interconnect elements in (multiple) the same tile or other tiles. Each programmable interconnect element 43 may also include connections to the interconnect segments 50 of the general routing resources between logic blocks (not shown). The general routing resources may include routing channels between logic blocks (not shown) and switch blocks (not shown), the logic blocks including tracks of interconnect segments (e.g., interconnect segments 50), and the switch blocks are used to connect the interconnect segments. The interconnect segments of the general routing resources (e.g., interconnect segments 50) may span one or more logic blocks. The programmable interconnect elements 43, together with the general routing resources, implement a programmable interconnect structure ("programmable interconnect") for the FPGA shown.

[0077] In one example implementation, the CLB 33 may include a configurable logic element (“CLE”) 44 and a single programmable interconnect element (“INT”) 43, the configurable logic element (“CLE”) 44 being programmable to implement user logic. In addition to one or more programmable interconnect elements, the BRAM 34 may also include a BRAM logic element (“BRL”) 45. Typically, the number of interconnect elements included in a tile depends on the height of the tile. In the illustrated example, the BRAM tile has the same height as five CLBs, but other numbers (e.g., four) may also be used. In addition to an appropriate number of programmable interconnect elements, the DSP block 35 may also include a DSP logic element (“DSPL”) 46. In addition to one instance of the programmable interconnect element 43, the IOB 36 may include, for example, two instances of an input / output logic element (“IOL”) 47. It will be clear to those skilled in the art that, for example, the actual IO pads connected to the IO logic element 47 are typically not limited to the area of ​​the input / output logic element 47.

[0078] In the illustrated example, the horizontal area near the center of the die (eg Figure 8 ) is used for configuration, clock and other control logic. Vertical columns 51 extending from this horizontal area or column are used to distribute clock and configuration signals across the width of the FPGA.

[0079] use Figure 8 Some FPGAs of the illustrated architecture include additional logic blocks that disrupt the conventional columnar structure that makes up the majority of the FPGA. The additional logic blocks can be programmable blocks and / or dedicated logic.

[0080] Notice, Figure 8The present invention is intended to illustrate only one exemplary FPGA architecture. For example, the number of logic blocks in a row, the relative width of the rows, the number and order of the rows, the types of logic blocks included in the rows, the relative sizes of the logic blocks, and the order of the rows. Figure 8 The interconnect / logic implementation included at the top is exemplary only. For example, in a real FPGA, more than one adjacent CLB row is typically included wherever a CLB appears to facilitate efficient implementation of user logic, but the number of adjacent CLB rows varies with the overall size of the FPGA.

[0081] In the above, reference is made to the embodiments presented in the present disclosure. However, the scope of the present disclosure is not limited to the specifically described embodiments. On the contrary, any combination of the described features and elements, whether or not related to different embodiments, is expected to implement and practice the intended embodiments. In addition, although the embodiments disclosed herein can achieve advantages over other possible solutions or prior art, whether a particular advantage is achieved by a given embodiment does not limit the scope of the present disclosure. Therefore, unless expressly stated in the claims, the foregoing aspects, features, embodiments and advantages are merely illustrative and are not considered elements or limitations of the appended claims.

[0082] As will be appreciated by those skilled in the art, the embodiments disclosed herein may be embodied as systems, methods, or computer program products. Thus, various aspects may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects (which may be collectively referred to herein as a "circuit," "module," or "system"). In addition, various aspects may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0083] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) would include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium is any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0084] A computer readable signal medium may include a propagated data signal in which (e.g., in baseband or as part of a carrier wave) a computer readable program code is embodied. Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0085] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0086] The computer program code for performing the operations of various aspects of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, Smalltalk, C++, etc. and conventional procedural programming languages, such as "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer by means of any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., by means of the Internet using an Internet service provider).

[0087] Various aspects of the present disclosure are described below with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products of the embodiments presented in the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device that produces a machine, so that instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the function / action specified in the flowchart and / or one or more block diagram frames.

[0088] These computer program instructions may also be stored in a computer-readable medium, which may direct a computer, other programmable data processing device, or other device to operate in a specific manner so that the instructions stored in the computer-readable medium produce an article of manufacture, which includes instructions for implementing the functions / actions specified in the flowchart and / or one or more block diagram boxes.

[0089] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0090] The flow chart and block diagram in the figure illustrate the possible architecture, function and operation of the system, method and computer program product according to various examples of the present invention.In this regard, each frame in the flow chart or block diagram can represent a part of a module, segment or instruction, and the instruction includes one or more executable instructions for realizing (multiple) specified logical functions.In some alternative implementations, the function marked in the frame may not occur in the order marked in the figure.For example, according to the function involved, the two frames shown in the sequence can actually be performed substantially at the same time, or the frame can sometimes be performed in the opposite order.It will also be noted that each frame of the block diagram and / or flow chart diagram and the combination of the frames in the block diagram and / or flow chart diagram can be realized by a system based on special-purpose hardware that performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

[0091] While the foregoing is directed to particular examples, other and further examples may be devised without departing from the basic scope thereof, and the scope of which is to be determined by the following claims.

Claims

1. An accelerator device, comprising: an interface configured to communicatively couple the accelerator device to a host; I / O logic, including a machine learning (ML) engine assigned to an I / O domain; as well as Coherence logic includes parameters corresponding to the ML algorithm, wherein the parameters are assigned to a coherence domain, wherein the parameters share the coherence domain with computing resources in the host. 2 . The accelerator device of claim 1 , wherein the interface is configured to use a coherent interconnect protocol to extend the coherence domain of the host into the accelerator device.

3. The accelerator device of claim 2, wherein the interface comprises an update agent configured to update the parameters in response to changes made in reference parameters stored in a memory associated with the host using a cache coherent shared memory multiprocessor paradigm.

4. The accelerator device according to claim 1, further comprising: A NoC is coupled to the I / O logic and the coherent logic, wherein at least one of the NoC, programmable logic (PL) to PL messaging, and wired signaling is configured to enable passing of the parameter from the coherent logic to the I / O logic. 5 . The accelerator device of claim 4 , wherein the ML engine is configured to process an ML data set received from the host using the parameters.

6. The accelerator device according to claim 1, further comprising: A programmable logic (PL) array, wherein a first plurality of PL blocks in the PL array are part of the I / O logic and are allocated to the I / O domain, and a second plurality of PL blocks in the PL array are part of the coherent logic and are allocated to the coherent domain.

7. The accelerator device according to claim 6, further comprising: a plurality of memory blocks, wherein a first subset of the plurality of memory blocks is part of the I / O logic and is allocated to the I / O domain, and a second subset of the plurality of memory blocks is part of the coherent logic and is allocated to the coherent domain, wherein the first subset of the plurality of memory blocks is capable of communicating with the first plurality of PL blocks but not directly with the second plurality of PL blocks, and the second subset of the plurality of memory blocks is capable of communicating with the second plurality of PL blocks but not directly with the first plurality of PL blocks.

8. An accelerator device, comprising: an interface configured to communicatively couple the accelerator device to a host; I / O logic, including a machine learning (ML) engine assigned to the I / O domain; as well as Coherence logic is configured to store updates for changing how the ML engine processes data, wherein the updates are assigned to a coherence domain, wherein the updates share the coherence domain with computing resources in the host. 9 . The accelerator device of claim 8 , wherein the interface is configured to use a coherent interconnect protocol to extend the coherence domain of the host into the accelerator device.

10. The accelerator device of claim 9, wherein the interface comprises an update agent configured to, in response to changes made in reference parameters stored in a memory associated with the host, store the updates using a cache coherent shared memory multiprocessor paradigm.

11. The accelerator device according to claim 8, further comprising: A NoC is coupled to the I / O logic and the coherent logic, wherein at least one of the NoC, programmable logic (PL) to PL messaging, and wired signaling is configured to enable the update to be passed from the coherent logic to the I / O logic. 12 . The accelerator device of claim 11 , wherein the ML engine is configured to process an ML dataset received from the host based on the update.

13. The accelerator device according to claim 8, further comprising: A programmable logic (PL) array, wherein a first plurality of PL blocks in the PL array are part of the I / O logic and are allocated to the I / O domain, and a second plurality of PL blocks in the PL array are part of the coherent logic and are allocated to the coherent domain.

14. The accelerator device according to claim 13, further comprising: a plurality of memory blocks, wherein a first subset of the plurality of memory blocks is part of the I / O logic and is allocated to the I / O domain, and a second subset of the plurality of memory blocks is part of the coherent logic and is allocated to the coherent domain, wherein the first subset of the plurality of memory blocks is capable of communicating with the first plurality of PL blocks but not directly with the second plurality of PL blocks, and the second subset of the plurality of memory blocks is capable of communicating with the second plurality of PL blocks but not directly with the first plurality of PL blocks.

15. An accelerator device, comprising: an interface configured to communicatively couple the accelerator device to a host storing a reference ML model; I / O logic, including a machine learning (ML) engine assigned to the I / O domain; as well as and a coherence logic configured to store a sub-portion of the reference ML model, wherein the sub-portion of the reference ML model shares a coherence domain with a computing resource in the host. 16 . The accelerator device of claim 15 , wherein the coherence logic is configured to store parameters corresponding to the sub-portion of the reference ML model, wherein the parameters are unique to the sub-portion of the reference ML model. 17 . The accelerator device of claim 15 , wherein the reference ML model in the host stores a plurality of ML models.

18. The accelerator device of claim 15, wherein the accelerator device is disposed in a container having a plurality of accelerator devices, wherein each accelerator device of the plurality of accelerator devices comprises a sub-portion of the reference ML model.

19. The accelerator device of claim 18, wherein the plurality of accelerator devices are sent different data sets to be processed using the sub-portion of the reference ML model.

20. The accelerator device of claim 18, wherein the plurality of accelerator devices are sent the same data set to be processed using the sub-portion of the reference ML model.

21. A system comprising: Multiple accelerator devices, each accelerator device includes: an interface configured to communicatively couple the accelerator device to one or more hosts; I / O hardware logic, including a machine learning (ML) engine assigned to an I / O domain; and coherence logic, including an ML model assigned to a coherence domain, wherein a first set of computing resources is managed in the I / O domain, and a second set of computing resources is managed in the coherence domain to coherently update the ML model; and A switch is configured to couple the plurality of accelerator devices to the one or more hosts.

22. The system of claim 21, wherein the switch is configured to couple a plurality of physical computing systems to the plurality of accelerator devices.

23. The system of claim 21, wherein the switch is in the coherence domain and the coherence logic is in each accelerator device of the plurality of accelerator devices.

24. The system of claim 23, wherein the switch is a cache layer between the coherent logic in each accelerator device of the plurality of accelerator devices and memory elements in the one or more hosts.

25. The system of claim 23, wherein when a reference ML model in the one or more hosts is updated, the switch is configured to send only an updated portion of the reference ML model to the coherent logic in the plurality of accelerator devices.

26. The system of claim 21, wherein the switch is configured to receive, from the one or more hosts, a different ML data set to be processed by an ML engine in each of the plurality of accelerator devices.

27. The system of claim 21, further comprising: An input / output expansion box includes the multiple accelerator devices and the switch.

28. An accelerator device, comprising: an interface configured to communicatively couple the accelerator device to a host; I / O hardware logic, including a machine learning (ML) engine assigned to the I / O domain; as well as A coherent logic comprising an ML model assigned to a coherent domain, wherein a first set of computing resources is managed in the I / O domain and a second set of computing resources is managed in the coherent domain to coherently update the ML model, and wherein the accelerator device is configured to provide different service levels for data traffic corresponding to the I / O logic and data traffic corresponding to the coherent logic.

29. The accelerator device of claim 28, wherein providing the different service levels comprises providing different qualities of service (QoS) for data traffic corresponding to the I / O logic and data traffic corresponding to the coherence logic.

30. The accelerator device of claim 28, wherein providing the different service levels comprises providing different bandwidths for the data traffic corresponding to the I / O logic and the data traffic corresponding to the coherence logic.

31. The accelerator device of claim 28, wherein providing the different service levels comprises providing different latencies for the data traffic corresponding to the I / O logic and the data traffic corresponding to the coherence logic.

32. The accelerator device of claim 28, further comprising: A network on chip (NoC) wherein a first portion of the NoC is part of the I / O domain and a second portion of the NoC is part of the coherence domain.

33. An accelerator device according to claim 32, wherein parameters of the NoC are configured to provide the different service levels, wherein the data traffic corresponding to the I / O logic and the data traffic corresponding to the coherent logic use the same hardware elements in the NoC but are treated differently by the same hardware elements based on the different service levels.

34. An accelerator device comprising: an interface configured to communicatively couple the accelerator device to one or more hosts; I / O hardware logic, including a machine learning (ML) engine assigned to the I / O domain; as well as Coherence logic is assigned to the coherence domain, wherein a first set of computing resources assigned to the I / O domain is allowed to communicate with a second set of computing resources assigned to the coherence domain to coherently update an ML model by using only connections internal to the accelerator device.

35. The accelerator device of claim 34, wherein communication between the first set of computing resources assigned to the I / O domain and the second set of computing resources assigned to the coherence domain can occur without data being routed through the host.

36. The accelerator device of claim 34, further comprising: A network on chip (NoC) wherein a first portion of the NoC is part of the I / O domain and a second portion of the NoC is part of the coherence domain.

37. The accelerator device of claim 36, wherein the NoC allows the first set of computing resources allocated to the I / O domain to communicate with the second set of computing resources allocated to the coherence domain.

38. The accelerator device of claim 34, further comprising: A first programmable logic (PL) block is provided in the I / O domain and a second PL block is provided in the coherent domain.

39. The accelerator device of claim 34, further comprising: A structure-to-structure connection between the first PL block and the second PL block.

40. The accelerator device of claim 39, wherein the fabric-to-fabric connection allows the first set of computing resources assigned to the I / O domain to communicate with the second set of computing resources assigned to the coherence domain.

Images