Efficient and parallel model execution

Abstract

An accelerator is disclosed. A circuit can process data to generate processed data. A first-tier storage can have a first capacity and a first latency. A second-tier storage can have a second capacity and a second latency. The second capacity can be greater than the first capacity, and the second latency can be slower than the first latency. A bus can be used to transfer at least one of data or processed data between the first-tier storage and the second-tier storage.

Description

Technical Field

[0001] The present disclosure relates generally to accelerators, and more particularly, to accelerators that can run models efficiently and in parallel. Background Art

[0002] Models, such as machine learning models, tend to be large. Some models may contain terabytes of data, and it is expected that model size may increase over time. Accelerators can run models, for example, to generate data for later machine learning analysis of that data.

[0003] But accelerators tend to have relatively limited memory and may not have enough memory capacity to store more than one model. Therefore, even if the accelerator supports enough processor cores to run multiple models in parallel, the models may be run sequentially, leaving some accelerator processing power unused.

[0004] There is still a need to use accelerators to improve the processing of the models. Summary of the Invention

[0005] Embodiments of the present disclosure include an accelerator. A circuit can process data. Two layers of storage providing different capacities and different latencies can be connected via a high-bandwidth bus to transfer data. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings described below are examples of how to implement embodiments of the present disclosure and are not intended to limit the embodiments of the present disclosure. Various embodiments of the present disclosure may include elements not shown in a particular drawing and / or may omit elements shown in a particular drawing. The accompanying drawings are intended to provide illustrations and may not be drawn to scale.

[0007] Figure 1 A machine including an accelerator supporting parallel execution of models according to an embodiment of the present disclosure is shown.

[0008] Figure 2 The embodiment according to the present disclosure is shown Figure 1 Details of the machine.

[0009] Figure 3 The embodiment according to the present disclosure is shown Figure 1 Details of the accelerator.

[0010] Figure 4 The embodiment according to the present disclosure is shown in Figure 3 Move batches of data between tiers.

[0011] Figure 5 The use of an embodiment according to the present disclosure is shown Figure 1Flowchart of an example process for running a model using an accelerator.

[0012] Figure 6 The embodiment according to the present disclosure is shown in Figure 3 Flowchart of an example process for transferring data between layers.

[0013] Figure 7 The embodiment according to the present disclosure is shown Figure 3 A flowchart of an example process of a prefetcher for prefetching data.

[0014] Figure 8 The embodiment according to the present disclosure is shown Figure 3 A flowchart of an example process for a prefetcher to prefetch batches of data.

[0015] Figure 9 The embodiment according to the present disclosure is shown Figure 1 The accelerator storage is composed of Figure 3 A flow chart of an example process for processing data by a circuit.

[0016] Figure 10 The embodiment according to the present disclosure is shown Figure 1 The host preprocessing should be Figure 1 A flowchart of an example process for using data in an accelerator.

[0017] Figure 11 The embodiment according to the present disclosure is shown Figure 1 Flowchart of an example process for preprocessing data with an accelerator.

[0018] Figure 12 The embodiment according to the present disclosure is shown Figure 1 The accelerator storage is composed of Figure 3 Flowchart of an example process for circuit preprocessing data.

[0019] Figure 13 The embodiment according to the present disclosure is shown Figure 1 The accelerator transfers data from Figure 3 The second-tier storage is loaded into Figure 3 Flowchart of an example process in a circuit. DETAILED DESCRIPTION

[0020] Reference will now be made in detail to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth to enable a thorough understanding of the present disclosure. However, it should be understood that one of ordinary skill in the art may practice the present disclosure without these specific details. In other cases, well-known methods, processes, components, circuits, and networks have not been described in detail to avoid unnecessarily obscuring various aspects of the embodiments.

[0021] It should be understood that although the terms first, second, etc. may be used to describe various elements in this document, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first module may be referred to as a second module, and similarly, a second module may be referred to as a first module without departing from the scope of this disclosure.

[0022] The terms used in the description of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. As used in the description of the present disclosure and the appended claims, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "and / or" used herein refer to and include any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms "including" and / or "comprising" when used in this specification specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. The components and features of the accompanying drawings are not necessarily drawn to scale.

[0023] Accelerators, which can include field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other processing hardware, can be used to run various models. One example of such a model is a machine learning (ML) model. By processing the ML model, the machine is able to recognize and process data that is later received. The accelerator can include multiple processing cores capable of processing multiple models in parallel.

[0024] But models can be large: some models may contain terabytes (TB) of data. On the other hand, the available memory in the accelerator can be relatively small and may be measured in gigabytes (GB) of data. It may be impossible to load an entire single model into the accelerator's memory, let alone load multiple models into the accelerator's memory to take advantage of parallel execution.

[0025] Furthermore, the model may be stored in a relatively large storage device, such as a solid-state drive (SSD) or hard disk drive. Such a storage device may be accessed via a bus, such as a Peripheral Component Interconnect Express (PCIe) bus. The bus may have a limit on the speed at which data can be transferred from the storage device, thereby creating a bottleneck for data to be transferred to the accelerator's memory.

[0026] Embodiments of the present disclosure address these issues by creating a two-tier accelerator. The accelerator can include accelerator memory as one tier and storage devices as another tier. High-bandwidth data transfer between the two tiers can bypass potentially bottlenecked buses (such as PCIe) and quickly move data to the accelerator memory for model execution.

[0027] Embodiments of the present disclosure can also fetch data into the accelerator memory in small batches. By processing small batches of data, multiple models can be run in parallel, thereby better utilizing the accelerator. Embodiments of the present disclosure can also include a prefetcher to prefetch data from the storage device into the accelerator memory in small batches, so that the accelerator does not have to wait to run the model.

[0028] Figure 1 A machine including an accelerator supporting parallel execution of models according to an embodiment of the present disclosure is shown. Figure 1 In the present invention, the machine 105 may also be referred to as a host or system, and may include a processor 110, a memory 115, and a storage device 120. The processor 110 may be a variety of processors. (For ease of illustration, the processor 110 and other components discussed below are shown as being external to the machine: embodiments of the present disclosure may include these components within the machine.) Although Figure 1 A single processor 110 is shown, but the machine 105 may include any number of processors, each of which may be a single-core processor or a multi-core processor, each of which may implement a reduced instruction set computer (RISC) architecture or a complex instruction set computer (CISC) architecture (among other possibilities), and which may be mixed in any desired combination.

[0029] The processor 110 may be coupled to a memory 115. The memory 115 may be a variety of memories, such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory, ferroelectric random access memory (FRAM), or non-volatile random access memory (NVRAM), such as magnetoresistive random access memory (MRAM), etc. As desired, the memory 115 may be a volatile memory or a non-volatile memory. The memory 115 may also be any desired combination of different memory types and may be managed by a memory controller 125. The memory 115 may be used to store what may be referred to as "short-term" data: that is, data that is not expected to be stored for an extended period of time. Examples of short-term data may include temporary files, data used locally by an application (which may have been copied from other storage locations), etc.

[0030] The processor 110 and the memory 115 may also support an operating system under which various applications may run. These applications may issue requests (also referred to as commands) to read data from or write data to any of the memories 115. When the storage device 120 is used to support applications that read or write data via a certain file system, a device driver 130 may be used to access the storage device 120. Figure 1 One storage device 120 is shown, but there may be any number (one or more) of storage devices in the machine 105. The storage device 120 may support any desired protocol or protocols, including, for example, the Non-Volatile Memory Express (NVMe) protocol.

[0031] although Figure 1 The general term "storage device" is used, but embodiments of the present disclosure may include any storage device format that can benefit from the use of computational storage units, examples of which may include hard disk drives and solid-state drives (SSDs). Any reference below to "SSDs" should be understood to include such other embodiments of the present disclosure.

[0032] The machine 105 may also include an accelerator 135 (which may also be referred to as a device). As described below, the accelerator 135 can support the operation of models (such as machine learning (ML) models) and can support the parallel operation of multiple models. The accelerator 135 can be implemented using any desired hardware. For example, the accelerator 135 or its components can be implemented using a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a data processing unit (DPU), a neural processing unit (NPU), a tensor processing unit (TPU), or a system on a chip (SoC), to name a few examples. The accelerator 135 can also be implemented using a combination of these elements.

[0033] The machine 105 may also include a network interface card 140 that can support a connection to a network 145. In addition to the data stored in the memory 115 and / or the storage device 120, some data to be used by the accelerator 135 can be accessed from the network interface card 140 or from a network address through the network 145. The network 145 can be a variety of networks, including a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or a global network such as the Internet. In addition, the network interface card 140 can support communication with the network 145 using a wired connection such as Ethernet, a wireless connection such as various wireless connections known as Wi-Fi, and various standards such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (which may include IEEE 802.11 a / b / g / n / ac / ax standards).

[0034] Figure 2 The embodiment according to the present disclosure is shown Figure 1 Details of the machine 105. Figure 2 In general, the machine 105 includes one or more processors 110, which may include a memory controller 120 and a clock 205, which may be used to coordinate the operation of the components of the machine. The processor 110 may also be coupled to a memory 115, which may include, for example, random access memory (RAM), read-only memory (ROM), or other state storage medium. The processor 110 may also be coupled to a storage device 125 and a network connector 210, which may be, for example, an Ethernet connector or a wireless connector. The processor 110 may also be connected to a bus 215, to which a user interface 220 and input / output (I / O) interface ports, which may be managed using an I / O engine 225, and other components may be attached.

[0035] Figure 3 The embodiment according to the present disclosure is shown Figure 1 Details of the accelerator 135. Figure 3 In the embodiment, the accelerator 135 may include circuit 305, a first layer of storage 310, and a second layer of storage 315. The circuit 305 may be considered as the processing center of the accelerator 135, and may perform the operation of various models. To this end, the circuit 305 may include cores 320-1 to 320-3 (which may be collectively referred to as cores 320). Similar to the cores in currently used processors and GPUs, the cores 320 may be parts that can run various instructions in the circuit 305. Each core 320 may run the kernel of an operating system and applications on top of that kernel: applications may run models. Each core 320 may operate independently of other cores. Some or all of the cores 320 may share some data, or each core 320 may use different data to operate. Some or all of the cores 320 may run the same instructions, or each core 320 may run different instructions. In addition, each core 320 may operate in parallel with the other cores 320: for example, operations in core 320-1 may not affect operations in core 320-2. Although Figure 3 Three cores 320 are shown, but embodiments of the present disclosure may include any number (one or more) of cores, limited only by what can be implemented with the available hardware.

[0036] The first tier of storage 310 and the second tier of storage 315 may be different storage options within the accelerator 135. For example, the first tier of storage 310 may be a form of memory similar to the memory 115 and may be implemented using DRAM or SRAM, among other options. When the first tier of storage 310 is implemented as a form of memory, the accelerator 135 may include a form of memory similar to the memory 115. Figure 1 A memory controller 325 of the memory controller 125 is used to manage writing data to and reading data from the first tier storage 310. Other options for the first tier storage 310 may include on-chip storage, such as a processor cache within the circuit 305 (which may be shared between the cores 320, or each core 320 may have its own processor cache) or a storage device such as a hard disk drive or SSD.

[0037] Second tier storage 315 may be similar to first tier storage 310. Generally, second tier storage 315 may be larger in capacity than first tier storage 310, but may have higher latency (i.e., may take longer to access data) than first tier storage 310. That is, second tier storage 315 may store more data than first tier storage 310, but accessing data from second tier storage 315 may take longer than accessing data from first tier storage 310. Like first tier storage 310, second tier storage 315 may be in the form of memory, a storage device such as a hard drive or SSD, or other form of storage. Note that in embodiments of the present disclosure where second tier storage 315 is a storage device, the term "data" may be omitted. Figure 1 storage device 120.

[0038] Connecting the first tier storage 310 and the second tier storage 315 may be a high bandwidth bus 330. The high bandwidth bus 330 may enable a large amount of data to be transferred between the first tier storage 310 and the second tier storage 315 quickly.

[0039] To understand the benefits of the high bandwidth bus 330, it is helpful to understand how data is transferred to the first tier of storage 310, and in particular how large amounts of data, such as machine learning models, can be transferred to the first tier of storage 310. Because such models can be large, they may be stored in Figure 1 Instead of being stored in memory 115, the data is stored on storage device 120. Figure 1 One way to transfer data between the storage device 120 and the first tier storage 310 is to have Figure 1 The storage device 120 reads the data into Figure 1As mentioned above, since the model may be large, the first layer storage 310 may only have enough space to store a single model, in which case only one model may be read into the storage device 120. Figure 1 The data can then be stored in the memory of the storage device 120. Figure 1 The memory of the storage device 120 is copied to the memory 115. The processor 110 can then access the data from the memory 115 and transfer the data to the first tier storage 310 via the bus connecting the processor 110 and the accelerator 135. Therefore, transferring the data to the first tier storage 310 may involve three copy operations: copying the data to Figure 1 A first copy operation in the memory of the storage device 120, a second copy operation to copy the data to the memory 115, and a third copy operation to copy the data to the first tier storage 310.

[0040] Furthermore, when data is copied to memory 115, it is important that the data remain in memory 115 so that it can be copied to first tier storage 310. However, in a system using virtual paging, some pages in memory 115 may be copied to the first tier storage 310. Figure 1 The processor 110 can then ensure that the pages to which the data is written are pinned memory: using pinned memory may further increase the latency of the overall process of copying data to the first tier storage 310.

[0041] As described above, because the models may be large, it may be possible to transfer only one model to the first tier storage 310. However, if there is only one model in the first tier storage 310, the circuit 305 may only process that model, eliminating the benefits of parallel model processing.

[0042] exist Figure 1 Another way to transfer data between the storage device 120 and the first tier storage 310 is to use Figure 1 The technology of directly transferring data between the storage device 120 and the first tier storage 310. Although this process may be faster than the above process, because it can eliminate one copy operation, the data can still be transferred through Figure 1The data is transferred over the PCIe bus between the storage device 120 and the first tier storage 310. The bandwidth limitations of the PCIe bus, coupled with the fact that the PCIe bus can be shared by other components, may limit the speed at which data can be transferred. Furthermore, even with this process, only one model may be transferred to the first tier storage 310 at a time, again eliminating the benefits of parallel model processing.

[0043] On the other hand, using a high-bandwidth bus 330 can allow more data to be transferred between the first tier of storage 310 and the second tier of storage 315 in a given time. The higher bandwidth can allow data to be transferred more quickly, thereby speeding up the execution of models. Data for more than one model can also be transferred in a given amount of time, thereby better utilizing the availability of cores 310. Furthermore, if the high-bandwidth bus 330 is dedicated to transferring data between the first tier of storage 310 and the second tier of storage 315, there is no need to worry about sending other data over the high-bandwidth bus 330, thereby limiting the amount of data that can be transferred between the first tier of storage 310 and the second tier of storage 315.

[0044] although Figure 3 Two tiers of storage 310 and 315 are shown, but embodiments of the present disclosure may include any number (one or more) of tiers of storage. For example, the accelerator 135 may include a third tier of storage that has a larger capacity than the second tier of storage 315 but is slower to access than the second tier of storage 315.

[0045] In addition to the circuit 305, the first tier storage 310, the second tier storage 315, the memory controller 325, and the high bandwidth bus 330, the accelerator 135 may also include a prefetcher 335. The prefetcher 335 may be used to prefetch data from the second tier storage 315 to the first tier storage 310 in anticipation of the data being needed. Figure 4 The operation of prefetcher 335 is discussed further.

[0046] In some embodiments of the present disclosure, the processor 110 may include a coordinator 340. The coordinator 340 may be used to coordinate the execution of models by the circuit 305. For example, the coordinator 340 may specify what data will be used next in the execution of the model and may arrange for the data to be transferred from the second-tier storage 315 to the first-tier storage 310. The coordinator 340 may also start the execution of the model in the circuit 305 when the data has been loaded into the first-tier storage 310. The coordinator 340 may use coordinator data 345 in the memory 115 to track the execution of various models and what data is being processed.

[0047] although Figure 3The coordinator 340 in the processor 110 and the coordinator data 345 in the memory 115 are shown, but embodiments of the present disclosure may allow the accelerator 135 to perform its own coordination. That is, instead of having the processor 110 coordinate the execution of the model, the accelerator 135 may use one (or more) cores 320 to perform the operations of the coordinator 340, and the first layer of storage 310 may store the coordinator data 345.

[0048] In some embodiments of the present disclosure, the accelerator 135 can process the data for the model immediately. However, in some embodiments of the present disclosure, it may be necessary to pre-process some of the data before the accelerator 135 can run the model. This pre-processing may be performed by the processor 110 or the circuit 305 (or one or more cores 320). The form that this pre-processing takes may depend on the data, and any form of pre-processing is intended to be covered by this disclosure: in fact, pre-processing can be considered to be another form of processing by the accelerator 135, except that the data can be processed more than once (once to complete the "pre-processing" and another time to process the "pre-processed" data). The processing itself can be thought of as taking input data d and pre-processing it to produce data d'.

[0049] The data in question may be located in a number of sources: memory 115, Figure 1 The memory in the network interface card 140, Figure 1 The network address on network 145, Figure 1 The storage device 120 or the first tier storage 310 or the second tier storage 315 etc. These different sources can be divided into two categories. Some sources (such as memory 115, Figure 1 The memory in the network interface card 140, Figure 1 The network address on network 145 or Figure 1 305 ), whereas data from sources external to the accelerator 135 may involve transferring the data into the accelerator 135. Note that regardless of whether the source is internal or external to the accelerator 135, the data in question may be stored on some physical device. For example, Figure 1 Data accessed by a network address on the network 145 may be stored on a device (memory, storage device, network interface card, etc.) to which the network address is assigned.

[0050] If the processor 110 is to perform preprocessing, the data to be transferred to the accelerator 135 may be preprocessed data, whereas if the accelerator 135 is to perform preprocessing and the data is not currently in the accelerator 135, the data (before preprocessing) may be transferred to the accelerator 135. In either case, some data will be transferred to the accelerator 135.

[0051] The accelerator 135 may support the use of a cache coherent interconnect protocol, such as the Compute Express Link (CXL) protocol, rather than using locked memory and input / output commands to transfer data between the processor 110 and the accelerator 135. The CXL protocol may provide protocols such as the CXL.mem protocol ( Figure 3 .mem 350) by which the processor 110 can directly access the first tier storage 310 (and possibly the second tier storage 315) within the accelerator 135 without using other input / output commands. In addition, the CXL protocol can provide a mechanism such as the CXL.cache protocol ( Figure 3 .cache 355) whereby the accelerator 135 can access data from memory 115 or other sources external to the accelerator 135 (such as Figure 1 The data in the storage device 120, Figure 1 The data in the memory of the network interface card 140 or Figure 1 For data from a network address on network 145 Figure 1 storage device 120, Figure 1 The memory in the network interface card 140 or Figure 1 Using a cache coherent interconnect protocol enables accelerator 135 to access data from a network address on network 145 without requiring the data to be placed in or manipulated by processor 110 or memory 115. In some embodiments of the present disclosure, the cache coherent interconnect protocol may also be used to access data in second-tier storage 315.

[0052] The use of a cache coherent interconnect protocol can support maintaining consistency between multiple versions of the same data: an update to the data in one location may cause the data in another location to be similarly updated if the data in memory 115 is also in first-tier storage 310. The use of a cache coherent interconnect protocol can provide more efficient transfer of data to the accelerator 135, regardless of the data in question or its source.

[0053] Figure 4 The embodiment according to the present disclosure is shown in Figure 3 Move batches of data between storage tiers 310 and 315. Figure 4 In the second storage layer 315, data for various models can be stored. Figure 4 , three models 405 - 1 , 405 - 2 , and 405 - 3 (which may be collectively referred to as models 405 ) are shown, but embodiments of the present disclosure may support any number (one or more) of models 405 .

[0054] Each model may include batches of data. For example, model 405-3 is shown as including batches 410-1, 410-2, and 410-3 (which may be collectively referred to as batch 410). Figure 4 Models 405-1 through 405-3 are shown as each including three batches 410 of data, but embodiments of the present disclosure may have models 405 including any number (one or more) of batches 410 of data. Furthermore, embodiments of the present disclosure may have a different number of batches 410 of data in each of the models 405: the models 405 need not have the same number of batches 410 of data.

[0055] The prefetcher 335 can identify Figure 3 The circuit 305 of the first layer storage 310 uses batches 410 in running the model 405. For example, the prefetcher 335 may know that the batches 410 can be used sequentially, and for a portion of a given run, each model 405 may only need one batch 410 of data. Therefore, instead of storing the entire model 405 in the first layer storage 310, individual batches 410 of the model 405 are stored in the first layer storage 310, and subsequent batches 410 are fetched when needed. Therefore, Figure 4 Three batches 410 are shown in the first tier storage 310. When the execution of batch 1A of model 405-1 is completed, batch 1B of model 405-1 can be loaded into the first tier storage 310; similarly, batches 410 of models 405-2 and 405-3 can be loaded into the first tier storage 310 as needed. The prefetcher 335 can use the high bandwidth bus 330 to load the batches 410 into the first tier storage 310. Thus, the prefetcher 335 can ensure that the batches 410 are loaded by the first tier storage 310. Figure 3 Data is available in each model 405 of the circuit 305 operation to maximize operating efficiency.

[0056] The prefetcher 335 can use hints to determine which batch 410 to load next into the first tier storage 310. For example, when designing a model, information about how to run the model can be provided to the prefetcher 335. The prefetcher 335 can then use these hints to select the next batch 410 of data to be loaded into the first tier storage 310.

[0057] Figure 5 The use of an embodiment according to the present disclosure is shown Figure 1 Flowchart of an example process for running a model using the accelerator 135. Figure 5 In block 505, Figure 1 The accelerator 135 can identify Figure 3 The second layer stores the data in 315 . Figure 3 The second tier storage 315 may have latency and capacity. At block 510, Figure 1 The accelerator 135 can transfer data from the Figure 3 The second layer storage 315 is transmitted to Figure 3 The first layer of storage 310. Figure 3 The first tier storage 310 may have latency and capacity: Figure 3 The capacity of the second tier storage 315 may be greater than the capacity of the first tier storage 310, but Figure 3 The latency of the second-tier storage 315 can also be better than Figure 3 The latency of the first tier of storage 310 is higher (i.e., slower). Finally, at block 515, Figure 3 The circuit 305 can process the data to generate processed data.

[0058] Figure 6 The embodiment according to the present disclosure is shown in Figure 3 Flowchart of an example process for transferring data between storage layers 310 and 315. Figure 6 There may be more than two Figure 3 tiers 310 and 315, and the data in question may be in the third tier of storage. Therefore, at block 605, Figure 1 The accelerator 135 can identify Figure 1 The third layer of the accelerator 135 stores the data, and at block 610, Figure 1 The accelerator 135 can transfer data from Figure 1 The third layer storage of the accelerator 135 is transmitted to Figure 3 The second tier of storage 315 allows the data to be later transferred to Figure 3 The first layer of storage 310. (Of course, Figure 1 The accelerator 135 can also transfer data directly from the third-tier storage to Figure 3 The first layer of storage 310. )

[0059] Figure 7 The embodiment according to the present disclosure is shown Figure 3 Flowchart of an example process of prefetching data by the prefetcher 335. Figure 7 In block 705, Figure 3 The prefetcher 335 can identify Figure 1 The accelerator 135 runs Figure 4 The data used when the model 405 is used (such as Figure 4 Then, at block 710, Figure 3The prefetcher 335 can transfer data from Figure 3 The second layer storage 315 pre-fetches Figure 3 The first layer of storage 310. Note that although Figure 7 Described Figure 3 The prefetcher 335 takes data from Figure 3 The second tier storage 315 is pre-fetched into the first tier storage 310 , but embodiments of the present disclosure may pre-fetch data from any storage tier and may store the pre-fetched data in any storage tier.

[0060] Figure 8 The embodiment according to the present disclosure is shown Figure 3 Flowchart of an example process for the prefetcher 335 to prefetch batches of data. Figure 8 In block 805, Figure 3 The prefetcher 335 can identify Figure 4 The data in the model 405 Figure 4 of batch 410. At block 810, Figure 3 The prefetcher 335 can prefetch for Figure 4 The data of the model 405 Figure 4 The first batch 410-1. At block 815, Figure 3 The prefetcher 335 can prefetch for Figure 4 The data of the model 405 Figure 4 The second batch 410-2. Note that it can be based on Figure 3 Does the prefetcher 335 anticipate that data will be needed soon? Figure 4 The second batch 410-2 performs block 815: For example, Figure 3 The prefetcher 335 can be based on Figure 3 Whether the circuit 305 has completed the data pre-fetched in block 810 Figure 4 The first batch 410 - 1 is processed to execute block 815 .

[0061] Figure 9 The embodiment according to the present disclosure is shown Figure 1 The accelerator 135 stores Figure 3 A flow chart of an example process for processing data by a circuit. Figure 9 can be considered as Figure 5 Continuation of the flowchart of the example process shown. Figure 9 In block 905, once Figure 3 The circuit 305 is completed as Figure 5 The data processing described in block 515, Figure 3 The accelerator 135 can then transfer the processed data from Figure 3 The first layer storage 310 is transmitted to Figure 3 The second layer of storage 315 (or transferred to Figure 1 any other storage tiers that may be available in the accelerator 135).

[0062] Figure 10 The embodiment according to the present disclosure is shown Figure 1 The host 105 preprocessing should be Figure 1 Flowchart of an example process for using data in the accelerator 135. Figure 10 In block 1005, Figure 1 The processor 110 can load data into Figure 1 The data may be loaded from any desired source: Figure 1 storage device 120, Figure 1 Network interface card 140, Figure 1 network address on network 145, or even from Figure 1 The accelerator 135 Figure 3 Layer 310 or 315 is loaded (although in the latter case, Figure 3 It may be more efficient for circuit 305 to process data if this is an option.) At block 1010, Figure 1 The processor 110 may process the data to generate processed data. Finally, at block 1015, Figure 1 The processor 110 can transmit the processed data to Figure 1 The accelerator 135 stores layers such as Figure 3 The processor 110 may use a cache coherent interconnect protocol (such as the CXL protocol) to transmit processed data to the first layer of storage 310. Figure 1 The accelerator 135 is stored in the layer.

[0063] Figure 11 The embodiment according to the present disclosure is shown Figure 1 Flowchart of an example process of preprocessing data by the accelerator 135. Figure 11 In block 1105, Figure 1 The accelerator 135 can load data from the source to Figure 3 The source can be, for example, Figure 1 Memory 115, Figure 1 storage device 120, Figure 1 Network interface card 140, Figure 1 The network address on the network 145 or Figure 3 The second layer of storage 315. Figure 1 The accelerator 135 may use a cache coherent interconnect protocol (such as the CXL protocol) to load data into Figure 3 The first layer of storage 310. In block 1110, Figure 3The circuit 305 can process the Figure 3 The first layer stores 310 data to generate processed data.

[0064] Figure 12 The embodiment according to the present disclosure is shown Figure 1 The accelerator 135 stores Figure 3 Flowchart of an example process for circuit preprocessing data. Figure 12 can be considered as Figure 11 Continuation of the flowchart of the example process shown. Figure 12 In block 1205, once Figure 3 The circuit 305 has been completed as Figure 11 The data processing described in block 1110, Figure 3 The accelerator 135 can be used in Figure 3 in the first layer of storage 310 (or in Figure 1 The processed data is stored in any other storage tier that may be available in the accelerator 135 .

[0065] Figure 13 The embodiment according to the present disclosure is shown Figure 1 The accelerator 135 transfers data from Figure 3 The second layer storage 315 is loaded into Figure 3 Flowchart of an example process in a circuit. Figure 13 In block 1305, Figure 1 The accelerator 135 can load data from the source to Figure 3 The first layer of storage 310. In block 1310, Figure 1 The accelerator 135 can transfer data from Figure 3 The first layer storage 310 is loaded into Figure 3 Circuit 305 (or Figure 3 is processed in core 320).

[0066] exist Figure 5-Figure 13 In the flowcharts, some embodiments of the present disclosure are shown. However, those skilled in the art will recognize that other embodiments of the present disclosure are possible by changing the order of blocks, omitting blocks, or including links not shown in the figures. All such variations of the flowcharts, whether explicitly described or not, are considered embodiments of the present disclosure.

[0067] Embodiments of the present disclosure include an accelerator. The accelerator may include two layers of storage, wherein a high-bandwidth bus connects the two layers of storage. Data can be moved between the two layers of storage via the high-bandwidth bus, thereby enabling rapid transfer of data for use by the accelerator's circuits and storage of unused data. By using a high-bandwidth bus between multiple layers of storage, embodiments of the present disclosure provide technical advantages over storing data on separate storage devices and transferring the data via a bus (such as a Peripheral Component Interconnect Express (PCIe) bus that may be shared or have lower bandwidth).

[0068] Embodiments of the present disclosure may also support the use of a cache coherent interconnect protocol, such as the Compute Express Link (CXL) protocol. By using a cache coherent interconnect protocol, data can be transferred from a source external to the accelerator to the accelerator using fewer operations, thereby providing a technical advantage of potentially faster data transfer to the accelerator.

[0069] In a solution based on Peripheral Component Interconnect Express (PCIe), multiple large models may be stored on solid-state drives (SSDs). Model data on the SSD can be cached in host dynamic random access memory (DRAM) before being copied to the accelerator DRAM. Even using a direct path between storage and the accelerator, where model data in the SSD can be copied directly to the accelerator DRAM, there may be significant data movement and kernel launch latency via PCIe. Because accelerator memory is limited, the CPU may coordinate model execution on the accelerator, running one model at a time.

[0070] When data is moved from the SSD to the accelerator, data movement and kernel launch latency using PCIe may occur. In addition, the CPU can coordinate model execution on the accelerator, running one model at a time. Parallel model execution in a PCIe-based solution may involve significant code changes.

[0071] Embodiments of the present disclosure can utilize the CXL tiered memory accelerator to achieve efficient and parallel kernel execution. The CXL tiered memory accelerator can directly access the data model stored in layer 2 (without involving the CPU or other technologies).

[0072] The CXL tiered memory accelerator provides higher internal bandwidth between tiers 1 and 2. Hot data is cached / moved to tier 1 via the prefetch module / auto-tiering module for faster access.

[0073] Embodiments of the present disclosure can reduce the amount of data movement and kernel boot latency. Compared to PCIe-based solutions, embodiments of the present disclosure can avoid intermediate data copies to the CPU and any bandwidth limitations associated with the PCIe bus. Because data can always be available locally, embodiments of the present disclosure can avoid putting kernel operations on hold. Embodiments of the present disclosure can use high internal bandwidth for data movement between layer 1 and layer 2. Overall data latency can be reduced from (DMA media +DMA SSD +DMA acc ) or (DMA media +DMA SsD ) is reduced to DMA media Because the accelerator can directly access Layer 1 and Layer 2, embodiments of the present disclosure can support seamless parallel model operation. Because the CPU can be avoided from being used for data replication, CPU utilization can be reduced. Embodiments of the present disclosure can provide improved end-to-end application performance.

[0074] A parallel model coordinator running on the host processor or in an accelerator coordinates the model execution and can pass programmer-provided hints for data placement / inter-tier movement to: place data in tier 1 or tier 2; prefetch data from tier 2 to tier 1; or statically partition tier 1.

[0075] In applications such as machine learning (ML), input data preprocessing may be required before kernel execution. The input data can come from a source such as a network. After running the preprocessing on the host or accelerator, the preprocessed data can be used by the accelerator to execute the kernel. In PCIe-based systems, additional data copying may be performed between the accelerator and host memory to copy the input data or preprocessed input to the accelerator memory. The data copying time overhead can significantly impact end-to-end performance.

[0076] Embodiments of the present disclosure can provide more efficient input data staging using a CXL Type 2 tiered memory accelerator. Embodiments of the present disclosure can support methods for efficient input data staging when preprocessing input data stored in different locations (network interface card (NIC), host central processing unit (CPU), dynamic random access memory (DRAM), Tier 2, etc.) on the host and on the accelerator.

[0077] Embodiments of the present disclosure can provide lower latency input staging between the host and accelerator DRAM for latency-sensitive use cases. This can also result in less data copying overall. Embodiments of the present disclosure can leverage higher internal bandwidth between Layer 1 and Layer 2 for capacity-hungry use cases. Embodiments of the present disclosure can improve end-to-end application performance.

[0078] The following discussion is intended to provide a brief, general description of one or more suitable machines in which certain aspects of the present disclosure may be implemented. The one or more machines may be controlled, at least in part, by input from traditional input devices (such as a keyboard, mouse, etc.) and by instructions received from another machine, interaction with a virtual reality (VR) environment, biometric feedback, or other input signals. As used herein, the term "machine" is intended to broadly encompass a single machine, a virtual machine, or a system of communicatively coupled machines, virtual machines, or devices operating together. Exemplary machines include computing devices (such as personal computers, workstations, servers, laptops, handheld devices, phones, tablets, etc.) and transportation devices, such as private or public transportation, e.g., cars, trains, taxis, etc.

[0079] One or more machines may include embedded controllers, such as programmable or non-programmable logic devices or arrays, application specific integrated circuits (ASICs), embedded computers, smart cards, and the like. One or more machines may utilize one or more connections to one or more remote machines, such as through a network interface, modem, or other communication coupling. The machines may be interconnected via physical and / or logical networks, such as an intranet, the Internet, a local area network, a wide area network, and the like. Those skilled in the art will appreciate that network communications may utilize various wired and / or wireless short-range or long-range carriers and protocols, including radio frequency (RF), satellite, microwave, Institute of Electrical and Electronics Engineers (IEEE) 802.11, Optical, infrared, cable, laser, etc.

[0080] Embodiments of the present disclosure may be described by reference to or in conjunction with associated data including functions, procedures, data structures, applications, etc., which, when accessed by a machine, cause the machine to perform tasks or define abstract data types or low-level hardware contexts. The associated data may be stored, for example, in volatile and / or non-volatile memory (e.g., RAM, ROM, etc.), or in other storage devices and their associated storage media (including hard drives, floppy disks, optical storage devices, magnetic tapes, flash memories, memory sticks, digital video disks, biometric storage devices, etc.). The associated data may be transmitted in the form of packets, serial data, parallel data, propagated signals, etc., in a transmission environment including a physical and / or logical network, and may be used in a compressed or encrypted format. The associated data may be used in a distributed environment and stored locally and / or remotely for machine access.

[0081] Embodiments of the present disclosure may include a tangible, non-transitory machine-readable medium including instructions executable by one or more processors, the instructions including instructions for performing elements of the present disclosure as described herein.

[0082] The various operations of the above methods may be performed by any suitable means capable of performing the operations (such as various hardware and / or software components, circuits and / or modules). Software may include an ordered list of executable instructions for implementing logical functions and may be contained in any "processor-readable medium" for use by or in conjunction with an instruction execution system, device or apparatus (such as a single-core processor or a multi-core processor or a system including a processor).

[0083] The blocks or steps of the methods or algorithms and functions described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, software modules executed by a processor, or a combination of the two. If implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a tangible, non-transitory computer-readable medium. The software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, removable disk, CD ROM, or any other form of storage medium known in the art.

[0084] The principles of the present disclosure have been described and illustrated with reference to the illustrated embodiments, and it will be appreciated that the illustrated embodiments may be modified in arrangement and detail without departing from these principles and may be combined in any desired manner. Furthermore, although the foregoing discussion has focused on specific embodiments, other configurations are also contemplated. Specifically, although expressions such as "according to embodiments of the present disclosure" are used herein, these phrases are intended to refer generally to the possibility of embodiments and are not intended to limit the present disclosure to specific embodiment configurations. As used herein, these terms may refer to the same embodiment or different embodiments that may be combined into other embodiments.

[0085] The foregoing illustrative embodiments should not be construed as limiting the disclosure herein. Although several embodiments have been described, it will be readily apparent to those skilled in the art that various modifications to these embodiments are possible without materially departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined in the claims.

[0086] The embodiments of the present disclosure can be extended to the following statements without limitation:

[0087] Statement 1. An embodiment of the present disclosure includes an accelerator comprising:

[0088] circuitry for processing data to produce processed data;

[0089] The first-tier storage includes the first capacity and the first latency;

[0090] A second-tier storage including a second capacity and a second latency, wherein the second capacity is greater than the first capacity and the second latency is slower than the first latency; and

[0091] A bus is configured to transfer at least one of data or processed data between the first tier storage and the second tier storage.

[0092] Statement 2. An embodiment of the present disclosure includes an accelerator according to Statement 1, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0093] Statement 3. Embodiments of the present disclosure include an accelerator according to Statement 1, wherein the circuit includes a first core and a second core.

[0094] Statement 4. An embodiment of the present disclosure includes an accelerator according to Statement 3, wherein:

[0095] The first tier storage is configured to store the data and the second data;

[0096] The first core is configured to process data to produce processed data; and

[0097] The second core is configured to process the second data to generate second processed data.

[0098] Statement 5. An embodiment of the present disclosure includes an accelerator according to Statement 4, wherein the first core and the second core are configured to operate in parallel.

[0099] Statement 6. An embodiment of the present disclosure comprises an accelerator according to Statement 1, wherein the first tier of storage is selected from the group consisting of dynamic random access memory (DRAM), static random access memory (SRAM), or processor cache.

[0100] Statement 7. An embodiment of the present disclosure comprises an accelerator according to Statement 1, wherein the second tier of storage is selected from the group consisting of DRAM, SRAM, or a storage device.

[0101] Statement 8. An embodiment of the present disclosure comprises an accelerator according to Statement 7, wherein the storage device is selected from the group consisting of a hard disk drive or a solid state drive (SSD).

[0102] Statement 9. An embodiment of the present disclosure includes an accelerator according to Statement 1, further comprising: a third layer of storage comprising a third capacity and a third latency, the third capacity being greater than the second capacity, and the third latency being slower than the second latency.

[0103] Statement 10. An embodiment of the present disclosure includes the accelerator of Statement 9, wherein the bus is configured to transfer the second data between the third tier storage and the second tier storage.

[0104] Statement 11. An embodiment of the present disclosure includes the accelerator according to Statement 10, wherein the bus is further configured to transfer third data between the third tier storage and the first tier storage.

[0105] Statement 12. An embodiment of the present disclosure comprises an accelerator according to Statement 1, wherein the bus comprises a bandwidth greater than a Peripheral Component Interconnect Express (PCIe) bus bandwidth.

[0106] Statement 13. An embodiment of the present disclosure comprises an accelerator according to Statement 1, further comprising a prefetcher.

[0107] Statement 14. An embodiment of the present disclosure includes an accelerator according to Statement 13, wherein the prefetcher is configured to transfer data from the second tier of storage to the first tier of storage via the bus.

[0108] Statement 15. An embodiment of the present disclosure includes an accelerator according to Statement 14, wherein:

[0109] The data includes a first batch of data and a second batch of data; and

[0110] The prefetcher is configured to transfer a first batch of data from the second tier storage to the first tier storage via the bus.

[0111] Statement 16. An embodiment of the present disclosure includes an accelerator according to Statement 15, wherein the prefetcher is configured to transfer the second batch of data from the second tier of storage to the first tier of storage based at least in part on the circuit processing the first batch of data.

[0112] Statement 17. An embodiment of the present disclosure includes an accelerator according to Statement 13, wherein the prefetcher is configured to select the data for transfer from the second tier storage to the first tier storage using a hint from the host.

[0113] Statement 18. An embodiment of the present disclosure includes the accelerator of Statement 1, wherein the bus is configured to transfer processed data from the first tier of storage to the second tier of storage.

[0114] Statement 19. An embodiment of the present disclosure includes an accelerator according to Statement 1, further comprising a coordinator.

[0115] Statement 20. An embodiment of the present disclosure comprises an accelerator according to Statement 19, wherein the coordinator is configured to identify data to be prefetched.

[0116] Statement 21. An embodiment of the present disclosure includes an accelerator according to Statement 20, wherein the coordinator is further configured to instruct the circuit to process the data.

[0117] Statement 22. An embodiment of the present disclosure includes an accelerator according to Statement 1, wherein the host comprises a coordinator.

[0118] Statement 23. An embodiment of the present disclosure includes a method comprising:

[0119] identifying data in a second-tier storage of the accelerator, the second-tier storage of the accelerator including a second capacity and a second latency;

[0120] transferring data from a second-tier storage of the accelerator to a first-tier storage of the accelerator via a bus, wherein the first-tier storage of the accelerator includes a first capacity and a first latency, wherein the second capacity is greater than the first capacity and the second latency is slower than the first latency; and

[0121] The data is processed in circuits of the accelerator to produce processed data.

[0122] Statement 24. An embodiment of the present disclosure includes a method according to Statement 23, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0123] Statement 25. Embodiments of the present disclosure include a method according to Statement 23, wherein:

[0124] Processing the data in the circuitry of the accelerator to produce processed data includes processing the data in a first core of the circuitry of the accelerator to produce processed data; and

[0125] The method further includes:

[0126] identifying second data in a second tier of storage of the accelerator;

[0127] transmitting the second data from the second tier storage of the accelerator to the first tier storage of the accelerator via the second bus; and

[0128] The second data is processed in a second core of the circuit of the accelerator to produce second processed data.

[0129] Statement 26. An embodiment of the present disclosure includes a method according to Statement 25, wherein processing the second data in the second core of the circuit of the accelerator to produce second processed data includes: processing the second data in the second core of the circuit of the accelerator to produce the second processed data in parallel with processing the data in the first core of the circuit of the accelerator to produce the processed data.

[0130] Statement 27. An embodiment of the present disclosure includes a method according to Statement 23, wherein the first tier of storage is selected from the group consisting of dynamic random access memory (DRAM), static random access memory (SRAM), or processor cache.

[0131] Statement 28. An embodiment of the present disclosure comprises a method according to Statement 23, wherein the second tier of storage is selected from the group consisting of DRAM, SRAM, or a memory device.

[0132] Statement 29. Embodiments of the present disclosure include the method of Statement 28, wherein the storage device is selected from the group consisting of a hard disk drive or a solid state drive (SSD).

[0133] Statement 30. An embodiment of the present disclosure includes a method according to Statement 23, further comprising:

[0134] identifying data in a third-tier storage of the accelerator, the third-tier storage comprising a third capacity and a third latency, the third capacity being greater than the second capacity, and the third latency being slower than the second latency; and

[0135] Transfer data from the accelerator's third-tier storage to the accelerator's second-tier storage.

[0136] Statement 31. An embodiment of the present disclosure includes a method according to Statement 30, wherein transferring data from the third tier storage of the accelerator to the second tier storage of the accelerator comprises transferring data from the third tier storage of the accelerator to the second tier storage of the accelerator over a bus.

[0137] Statement 32. An embodiment of the present disclosure comprises a method according to Statement 30, wherein the bus comprises a bandwidth greater than a Peripheral Component Interconnect Express (PCIe) bus bandwidth.

[0138] Statement 33. An embodiment of the present disclosure includes the method of Statement 23, wherein transferring data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over the bus comprises prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator.

[0139] Statement 34. An embodiment of the present disclosure includes a method according to Statement 33, wherein prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator comprises prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over a bus.

[0140] Statement 35. An embodiment of the present disclosure includes a method according to Statement 33, wherein prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator comprises:

[0141] identifying a first batch of data in the data and a second batch of data in the data; and

[0142] Prefetch the first batch of data from the accelerator's second-tier storage to the accelerator's first-tier storage.

[0143] Statement 36. An embodiment of the present disclosure includes a method according to Statement 35, wherein prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator further comprises prefetching a second batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator based at least in part on the first batch of data being processed by circuitry of the accelerator.

[0144] Statement 37. An embodiment of the present disclosure includes a method according to Statement 33, wherein prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator comprises prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator based at least in part on a hint from the host.

[0145] Statement 38. An embodiment of the present disclosure includes a method according to Statement 33, wherein prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator comprises prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator based at least in part on a signal from the coordinator.

[0146] Statement 39. An embodiment of the present disclosure includes the method of Statement 23, further comprising transferring the processed data from the first tier of storage of the accelerator to the second tier of storage of the accelerator.

[0147] Statement 40. Embodiments of the present disclosure include a method according to Statement 23, wherein:

[0148] Transferring the data from the second tier of storage of the accelerator to the first tier of storage of the accelerator via the bus includes: transferring the data from the second tier of storage of the accelerator to the first tier of storage of the accelerator via the bus based at least in part on the first signal from the coordinator; and

[0149] Processing the data in the circuit of the accelerator to produce processed data includes processing the data in the circuit of the accelerator to produce processed data based at least in part on the second signal from the coordinator.

[0150] Statement 41. Embodiments of the present disclosure include a method according to Statement 40, wherein the accelerator comprises a coordinator.

[0151] Statement 42. Embodiments of the present disclosure include a method according to Statement 40, wherein the host comprises a coordinator.

[0152] Statement 43. An embodiment of the present disclosure comprises an article comprising a non-transitory storage medium having stored thereon instructions that, when executed by a machine, cause:

[0153] identifying data in a second-tier storage of the accelerator, the second-tier storage of the accelerator comprising a second capacity and a second latency;

[0154] transferring data from a second-tier storage of the accelerator to a first-tier storage of the accelerator via a bus, wherein the first-tier storage of the accelerator includes a first capacity and a first latency, wherein the second capacity is greater than the first capacity and the second latency is slower than the first latency; and

[0155] The data is processed in circuits of the accelerator to produce processed data.

[0156] Statement 44. An embodiment of the present disclosure includes an article according to statement 43, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0157] Statement 45. Embodiments of the present disclosure include an article according to Statement 43, wherein:

[0158] Processing the data in the circuitry of the accelerator to produce processed data includes processing the data in a first core of the circuitry of the accelerator to produce processed data; and

[0159] The non-transitory storage medium has stored thereon further instructions that, when executed by the machine, cause:

[0160] identifying second data in a second tier storage of the accelerator;

[0161] transmitting the second data from the second tier storage of the accelerator to the first tier storage of the accelerator via the second bus; and

[0162] The second data is processed in a second core of the circuit of the accelerator to produce second processed data.

[0163] Statement 46. An embodiment of the present disclosure includes an article according to Statement 45, wherein processing the second data in the second core of the circuit of the accelerator to produce second processed data includes: processing the second data in the second core of the circuit of the accelerator to produce the second processed data in parallel with processing the data in the first core of the circuit of the accelerator to produce the processed data.

[0164] Statement 47. An embodiment of the present disclosure includes an article according to Statement 43, wherein the first tier of storage is selected from the group consisting of dynamic random access memory (DRAM), static random access memory (SRAM), or processor cache.

[0165] Statement 48. An embodiment of the present disclosure comprises an article according to Statement 43, wherein the second tier of storage is selected from the group consisting of DRAM, SRAM, or a memory device.

[0166] Statement 49. Embodiments of the present disclosure include an article according to Statement 48, wherein the storage device is selected from the group consisting of a hard disk drive or a solid state drive (SSD).

[0167] Statement 50. Embodiments of the present disclosure include an article according to Statement 43, the non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause:

[0168] identifying data in a third-tier storage of the accelerator, the third-tier storage comprising a third capacity and a third latency, the third capacity being greater than the second capacity, and the third latency being slower than the second latency; and

[0169] Transfer data from the accelerator's third-tier storage to the accelerator's second-tier storage.

[0170] Statement 51. An embodiment of the present disclosure includes an article according to Statement 50, wherein transferring data from the third tier storage of the accelerator to the second tier storage of the accelerator comprises transferring data from the third tier storage of the accelerator to the second tier storage of the accelerator over a bus.

[0171] Statement 52. An embodiment of the present disclosure includes an article according to Statement 50, wherein the bus comprises a bandwidth greater than a Peripheral Component Interconnect Express (PCIe) bus bandwidth.

[0172] Statement 53. An embodiment of the present disclosure includes the article of Statement 43, wherein transferring data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over the bus comprises prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator.

[0173] Statement 54. An embodiment of the present disclosure includes an article according to Statement 53, wherein prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator comprises prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over a bus.

[0174] Statement 55. An embodiment of the present disclosure includes the article of Statement 53, wherein prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator comprises:

[0175] identifying a first batch of data in the data and a second batch of data in the data; and

[0176] Prefetch the first batch of data from the accelerator's second-tier storage to the accelerator's first-tier storage.

[0177] Statement 56. An embodiment of the present disclosure includes an article according to statement 55, wherein prefetching data from the second tier of storage of the accelerator to the first tier of storage of the accelerator further comprises prefetching a second batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator based at least in part on the first batch of data being processed by circuitry of the accelerator.

[0178] Statement 57. An embodiment of the present disclosure includes an article according to Statement 53, wherein prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator comprises prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator based at least in part on a hint from the host.

[0179] Statement 58. An embodiment of the present disclosure includes a method according to Statement 53, wherein prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator comprises prefetching data from the second tier storage of the accelerator to the first tier storage of the accelerator based at least in part on a signal from the coordinator.

[0180] Statement 59. An embodiment of the present disclosure comprises the article of statement 43, the non-transitory storage medium having stored thereon further instructions that, when executed by the machine, cause the processed data to be transferred from the first tier of storage of the accelerator to the second tier of storage of the accelerator.

[0181] Statement 60. Embodiments of the present disclosure include an article according to Statement 43, wherein:

[0182] Transferring the data from the second tier storage of the accelerator to the first tier storage of the accelerator via the bus includes: transferring the data from the second tier storage of the accelerator to the first tier storage of the accelerator via the bus based at least in part on the first signal from the coordinator; and

[0183] Processing the data in the circuit of the accelerator to produce processed data includes processing the data in the circuit of the accelerator to produce processed data based at least in part on the second signal from the coordinator.

[0184] Statement 61. An embodiment of the present disclosure includes an article according to Statement 60, wherein the accelerator comprises a coordinator.

[0185] Statement 62. An embodiment of the present disclosure includes article according to Statement 60, wherein the host comprises a coordinator.

[0186] Statement 63. Embodiments of the present disclosure include a system comprising:

[0187] host memory, which stores data;

[0188] a host processor for processing the data to generate processed data; and

[0189] Accelerators, including:

[0190] circuitry that processes the processed data; and

[0191] Tier storage, which is used to store processed data;

[0192] The host processor may transmit the processed data to the tier storage of the accelerator using a cache coherent interconnect protocol.

[0193] Statement 64. An embodiment of the present disclosure includes a system according to statement 63, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0194] Statement 65. An embodiment of the present disclosure includes a system according to Statement 63, wherein the host processor is configured to transfer the processed data to the tier storage of the accelerator without storing the processed data in the locked memory of the host memory.

[0195] Statement 66. An embodiment of the present disclosure comprises a system according to Statement 63, wherein the cache coherent interconnect protocol comprises a Compute Express Link (CXL) protocol.

[0196] Statement 67. An embodiment of the present disclosure includes a system according to Statement 63, wherein the host processor can transfer processed data to the accelerator without using a Peripheral Component Interconnect Express (PCIe) bus.

[0197] Statement 68. Embodiments of the present disclosure include an accelerator comprising:

[0198] Tier storage; and

[0199] circuitry that processes the data to generate processed data and stores the processed data in the layer storage,

[0200] The accelerator is configured to load data from the device using a cache coherent interconnect protocol.

[0201] Statement 69. An embodiment of the present disclosure includes an accelerator according to Statement 68, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0202] Statement 70. An embodiment of the present disclosure comprises an accelerometer according to Statement 68, wherein the device is external to the accelerometer.

[0203] Statement 71. An embodiment of the present disclosure comprises an accelerator according to Statement 68, wherein the device is selected from the group consisting of host memory, a network interface card, a network address, or a second layer of storage.

[0204] Statement 72. An embodiment of the present disclosure comprises an accelerator according to Statement 71, wherein the tier of storage is selected from the group consisting of DRAM, SRAM, hard disk drive, or solid state drive (SSD).

[0205] Statement 73. An embodiment of the present disclosure comprises an accelerator according to Statement 68, wherein the cache coherent interconnect protocol comprises a Compute Express Link (CXL) protocol.

[0206] Statement 74. An embodiment of the present disclosure comprises an accelerator according to Statement 68, wherein the circuit comprises a first core and a second core.

[0207] Statement 75. An embodiment of the present disclosure includes an accelerator according to Statement 74, wherein:

[0208] The first core is configured to process data to produce processed data; and

[0209] The second core is configured to process the second data to generate second processed data.

[0210] Statement 76. An embodiment of the present disclosure comprises an accelerator according to Statement 75, wherein the first core and the second core are configured to operate in parallel.

[0211] Statement 77. An embodiment of the present disclosure comprises an accelerator according to Statement 68, wherein the tier of storage is selected from the group consisting of dynamic random access memory (DRAM), static random access memory (SRAM), or processor cache.

[0212] Statement 78. An embodiment of the present disclosure includes an accelerator according to Statement 68, wherein:

[0213] The accelerator also includes a memory controller; and

[0214] The circuit is configured to store the processed data in the tier storage using the memory controller.

[0215] Statement 79. An embodiment of the present disclosure includes an accelerator according to Statement 68, wherein:

[0216] The accelerator is configured to load data from the device into the tier storage using a cache coherent interconnect protocol; and

[0217] The circuitry of the accelerator is configured to access data from the tier storage of the accelerator.

[0218] Statement 80. An embodiment of the present disclosure comprises an accelerator according to Statement 68, wherein the accelerator is configured to load data from the device into the circuit using a cache coherent interconnect protocol.

[0219] Statement 81. An embodiment of the present disclosure includes a method comprising:

[0220] Loading data from host memory into the host processor;

[0221] processing the data using a host processor to produce processed data; and

[0222] Processed data is delivered to the accelerator's tier storage using a cache-coherent interconnect protocol.

[0223] Statement 82. An embodiment of the present disclosure includes a method according to Statement 81, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0224] Statement 83. An embodiment of the present disclosure includes a method according to Statement 81, wherein transferring the processed data to the tier storage of the accelerator using a cache coherent interconnect protocol includes transferring the processed data to the tier storage of the accelerator without storing the processed data in a locked memory of the host memory.

[0225] Statement 84. An embodiment of the present disclosure comprises a method according to Statement 81, wherein the cache coherent interconnect protocol comprises a Compute Express Link (CXL) protocol.

[0226] Statement 85. An embodiment of the present disclosure includes a method according to Statement 81, wherein transferring the processed data to the tier storage of the accelerator using a cache coherent interconnect protocol includes transferring the processed data to the tier storage of the accelerator without using a peripheral component interconnect express (PCIe) bus.

[0227] Statement 86. An embodiment of the present disclosure includes a method comprising:

[0228] Loading data from the device into the accelerator's tier storage using a cache-coherent interconnect protocol; and

[0229] Data from the tier storage is processed using circuitry of the accelerator to produce processed data.

[0230] Statement 87. An embodiment of the present disclosure includes a method according to Statement 86, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0231] Statement 88. An embodiment of the present disclosure comprises a method according to Statement 86, wherein the device is selected from the group consisting of host memory, a network interface card, a network address, or a second layer storage.

[0232] Statement 89. An embodiment of the present disclosure includes a method according to Statement 86, wherein loading data from a device into the tier storage of the accelerator using a cache coherent interconnect protocol comprises loading data from a device external to the accelerator into the tier storage of the accelerator using a cache coherent interconnect protocol.

[0233] Statement 90. An embodiment of the present disclosure comprises a method according to Statement 86, wherein the tier of storage is selected from the group consisting of DRAM, SRAM, hard disk drive, or solid state drive (SSD).

[0234] Statement 91. An embodiment of the present disclosure comprises the method of Statement 86, wherein the cache coherent interconnect protocol comprises a Compute Express Link (CXL) protocol.

[0235] Statement 92. Embodiments of the present disclosure include a method according to Statement 86, wherein:

[0236] Loading data from the device into the accelerator's tier storage using a cache-coherent interconnect protocol involves:

[0237] Load data into the accelerator's layer storage;

[0238] loading the second data into a tier storage of a circuit of the accelerator; and

[0239] Processing data using circuits of an accelerator to produce processed data includes:

[0240] processing the data using a first core of the circuit of the accelerator to generate processed data; and

[0241] The second data is processed using a second core of the circuit of the accelerator to generate second processed data.

[0242] Statement 93. An embodiment of the present disclosure includes a method according to Statement 92, wherein processing the second data using the second core of the circuit of the accelerator to produce second processed data includes: processing the second data using the second core of the circuit of the accelerator to produce the second processed data in parallel with processing the data using the first core of the circuit of the accelerator to produce the processed data.

[0243] Statement 94. An embodiment of the present disclosure comprises a method according to Statement 86, wherein the tier of storage is selected from the group consisting of dynamic random access memory (DRAM), static random access memory (SRAM), or processor cache.

[0244] Statement 95. An embodiment of the present disclosure includes the method of Statement 86, further comprising storing, by the circuitry of the accelerator, the processed data in a tier storage of the accelerator.

[0245] Statement 96. An embodiment of the present disclosure includes a method according to Statement 95, wherein storing, by the circuitry of the accelerator, the processed data in the tier storage of the accelerator comprises storing, by the circuitry of the accelerator, the processed data in the tier storage of the accelerator using a memory controller of the accelerator.

[0246] Statement 97. An embodiment of the present disclosure comprises a method according to Statement 86, wherein loading data from the device into the tier storage of the accelerator using a cache coherent interconnect protocol comprises loading data from the tier storage of the accelerator into circuitry of the accelerator.

[0247] Statement 98. An embodiment of the present disclosure comprises an article comprising a non-transitory storage medium having stored thereon instructions that, when executed by a machine, cause:

[0248] Loading data from host memory into the host processor;

[0249] processing the data using a host processor to produce processed data; and

[0250] Processed data is delivered to the accelerator's tier storage using a cache-coherent interconnect protocol.

[0251] Statement 99. An embodiment of the present disclosure includes an article according to statement 98, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0252] Statement 100. An embodiment of the present disclosure comprises an article according to statement 98, wherein transferring the processed data to the tier storage of the accelerator using a cache coherent interconnect protocol comprises transferring the processed data to the tier storage of the accelerator without storing the processed data in a locked memory of the host memory.

[0253] Statement 101. An embodiment of the present disclosure comprises article according to Statement 98, wherein the cache coherent interconnect protocol comprises a Compute Express Link (CXL) protocol.

[0254] Statement 102. An embodiment of the present disclosure includes an article according to statement 98, wherein transferring the processed data to the tier storage of the accelerator using a cache coherent interconnect protocol includes transferring the processed data to the tier storage of the accelerator without using a peripheral component interconnect express (PCUe) bus.

[0255] Statement 103. An embodiment of the present disclosure comprises an article comprising a non-transitory storage medium having stored thereon instructions that, when executed by a machine, cause:

[0256] Loading data from the device into the accelerator's tier storage using a cache-coherent interconnect protocol; and

[0257] Data from the tier storage is processed using circuitry of the accelerator to produce processed data.

[0258] Statement 104. An embodiment of the present disclosure includes an article according to statement 103, wherein the accelerator is implemented using at least one of a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), a tensor processing unit (TPU), a neural processing unit (NPU), or a system on a chip (SoC).

[0259] Statement 105. An embodiment of the present disclosure comprises an article according to statement 103, wherein the device is selected from the group consisting of host memory, a network interface card, a network address, or a second layer storage.

[0260] Statement 106. An embodiment of the present disclosure includes an article according to statement 103, wherein loading data from a device into the tier storage of the accelerator using a cache coherent interconnect protocol comprises loading data from a device external to the accelerator into the tier storage of the accelerator using a cache coherent interconnect protocol.

[0261] Statement 107. Embodiments of the present disclosure include article according to statement 103, wherein the tier of storage is selected from the group consisting of DRAM, SRAM, hard disk drive, or solid state drive (SSD).

[0262] Statement 108. An embodiment of the present disclosure comprises article according to Statement 103, wherein the cache coherent interconnect protocol comprises a Compute Express Link (CXL) protocol.

[0263] Statement 109. Embodiments of the present disclosure include an article according to Statement 103, wherein:

[0264] Loading data from the device into the accelerator's tier storage using a cache-coherent interconnect protocol involves:

[0265] Load data into the accelerator's layer storage;

[0266] loading the second data into a tier storage of a circuit of the accelerator; and

[0267] Processing data using circuits of an accelerator to produce processed data includes:

[0268] processing the data using a first core of a circuit of an accelerator to produce processed data; and

[0269] The second data is processed using a second core of the circuit of the accelerator to generate second processed data.

[0270] Statement 110. An embodiment of the present disclosure includes an article according to statement 109, wherein processing the second data using the second core of the circuit of the accelerator to produce the second processed data includes: processing the second data using the second core of the circuit of the accelerator to produce the second processed data in parallel with processing the data using the first core of the circuit of the accelerator to produce the processed data.

[0271] Statement 111. An embodiment of the present disclosure comprises an article according to statement 103, wherein the tier of storage is selected from the group consisting of dynamic random access memory (DRAM), static random access memory (SRAM), or processor cache.

[0272] Statement 112. An embodiment of the present disclosure comprises an article according to statement 103, the non-transitory storage medium having stored thereon further instructions that, when executed by the machine, cause circuitry of the accelerator to store processed data in tier storage of the accelerator.

[0273] Statement 113. An embodiment of the present disclosure includes an article according to Statement 112, wherein storing, by the circuitry of the accelerator, the processed data in the tier storage of the accelerator comprises storing, by the circuitry of the accelerator, the processed data in the tier storage of the accelerator using a memory controller of the accelerator.

[0274] Statement 114. An embodiment of the present disclosure includes article according to statement 103, wherein loading data from the device into the tier storage of the accelerator using the cache coherent interconnect protocol comprises loading data from the tier storage of the accelerator into circuitry of the accelerator.

[0275] Therefore, in view of the various arrangements of the embodiments described herein, this detailed description and accompanying material are intended to be illustrative only and should not be taken as limiting the scope of the disclosure. Therefore, the disclosure claims all such modifications that may fall within the scope and spirit of the appended claims and their equivalents.

Claims

1. An accelerator, comprising: circuitry that executes instructions on a first batch of data to produce a first processed batch of data, the data model comprising the first batch of data and a second batch of data, the circuitry comprising one or more cores, the circuitry configured to execute the instructions on the data model using the first batch of data and the second batch of data; A first-tier storage having a first capacity and a first latency; The second-tier storage has a second capacity and a second latency, wherein the second capacity is greater than the first capacity and the second latency is slower than the first latency; a bus to transfer at least one of the first batch of data or the first processed batch of data between the first tier of storage and the second tier of storage; as well as A prefetcher transfers the first data batch from the second tier storage to the first tier storage via the bus based at least in part on data placement information regarding the first data batch from the host.

2. The accelerator according to claim 1, wherein The bus includes a bandwidth greater than a Peripheral Component Interconnect Express (PCIe) bus bandwidth.

3. The accelerator according to claim 1, wherein: The circuit includes a first core and a second core.

4. The accelerator according to claim 3, wherein: The first tier of storage is configured to store a first batch of data and a second batch of data; The first core is configured to process a first data batch to produce a first processed data batch; and The second core is configured to process a second batch of data to produce a second processed batch of data.

5. The accelerator according to claim 1, wherein: The prefetcher is configured to transfer a second batch of data from the second tier of storage to the first tier of storage via the bus.

6. The accelerator according to claim 5, wherein The prefetcher is configured to transfer a second batch of data from the second tier of storage to the first tier of storage based at least in part on the circuitry executing the instruction on the first batch of data. The accelerator of claim 1 , further comprising a coordinator configured to identify a first batch of data for prefetching.

8. The accelerator according to claim 1, wherein The coordinator is further configured to instruct the circuit to process a first data batch.

9. A method comprising: identifying, based at least in part on data placement information about the first data batch from the host, the first data batch in a second tier of storage of the accelerator, the second tier of storage of the accelerator having a second capacity and a second latency, the second tier of storage storing a data model including the first data batch and the second data batch; transferring a first data batch from a second-tier storage of the accelerator to a first-tier storage of the accelerator via a bus, wherein the first-tier storage of the accelerator has a first capacity and a first latency, wherein the second capacity is greater than the first capacity and the second latency is slower than the first latency; as well as executing instructions in circuitry of the accelerator on a first batch of data to produce a first processed batch of data, wherein the circuit comprises one or more cores, and The circuitry is configured to execute the instructions on the data model using a first batch of data and a second batch of data.

10. The method according to claim 9, wherein: Transferring the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over the bus includes prefetching the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator.

11. The method according to claim 10, wherein: Prefetching a first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator includes prefetching a second batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator.

12. The method according to claim 11, wherein Prefetching the second batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator further includes prefetching the second batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator based at least in part on executing the instruction on the first batch of data by the circuit of the accelerator.

13. The method according to claim 10, wherein: Prefetching a first batch of data from the second tier storage of the accelerator to the first tier storage of the accelerator includes prefetching the first batch of data from the second tier storage of the accelerator to the first tier storage of the accelerator based at least in part on the data placement information from the host.

14. The method according to claim 10, wherein: Prefetching the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator includes prefetching the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator based at least in part on a signal from a coordinator.

15. The method of claim 9, wherein: Transferring the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over the bus comprises transferring the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over the bus based at least in part on a first signal from the coordinator; and Executing the instructions on the first batch of data in the circuitry of the accelerator to produce a first processed batch of data includes executing the instructions on the first batch of data in the circuitry of the accelerator to produce a first processed batch of data based at least in part on a second signal from a coordinator.

16. An article comprising a non-transitory storage medium having stored thereon instructions that, when executed by a machine, cause: identifying, based at least in part on data placement information about the first data batch from the host, the first data batch in a second tier of storage of the accelerator, the second tier of storage of the accelerator having a second capacity and a second latency, the second tier of storage storing a data model including the first data batch and the second data batch; transferring a first batch of data from a second tier of storage of the accelerator to a first tier of storage of the accelerator via a bus, wherein the first tier of storage of the accelerator has a first capacity and a first latency, wherein the second capacity is greater than the first capacity and the second latency is slower than the first latency; and executing instructions in circuitry of the accelerator on a first batch of data to produce a first processed batch of data, wherein the circuit comprises one or more cores, and The circuitry is configured to execute the instructions on the data model using a first batch of data and a second batch of data.

17. The article of claim 16, wherein Transferring the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator over the bus includes prefetching the first batch of data from the second tier of storage of the accelerator to the first tier of storage of the accelerator.

18. The article of claim 16, the non-transitory storage medium having stored thereon further instructions that, when executed by a machine, cause a first processed batch of data to be transferred from a first tier of storage of the accelerator to a second tier of storage of the accelerator.

Images