System and method for processing tasks

Abstract

Systems and methods for processing tasks are disclosed. The processing circuitry is configured to: identify first data associated with a task, the task associated with a machine learning program; determining a first attribute associated with the first data; providing the first data to the first device based on determining the first attribute; identifying second data associated with the task, the task associated with a machine learning program; determining a second attribute associated with the second data; and providing the second data to a second device different from the first device, where the machine learning program is configured to generate a result based on the input.

Description

[0001] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 535,536, filed on August 30, 2023, entitled “HETEROGENEOUS MEMORY SYSTEM FOR AI PIPELINES,” and U.S. Application No. 18 / 378,958, filed on October 11, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] One or more aspects of embodiments according to the present disclosure relate to a storage system, and more particularly, to a system and method for processing one or more tasks. Background Art

[0003] The use of artificial intelligence (AI) has increased dramatically in the past few years. AI has become commonly used in areas such as image classification, speech recognition, media analysis, health care, autonomous machines, smart assistants, etc. Training AI models often requires processing and storing large amounts (e.g., terabytes) of data.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art. Summary of the invention

[0005] One or more embodiments of the present disclosure are directed to a system, the system comprising: a first storage medium; and a processing circuit in communication with the first storage medium. The processing circuit is configured to: identify first data associated with a task, the task being associated with a machine learning program; determine a first attribute associated with the first data; provide the first data to a first device based on the determination of the first attribute; identify second data associated with the task, the task being associated with the machine learning program; determine a second attribute associated with the second data; and provide the second data to a second device different from the first device, wherein the machine learning program is configured to generate a result based on an input.

[0006] In some embodiments, the first device includes at least one of a graphics processor, a tensor processor, and a coprocessor.

[0007] In some embodiments, the second device includes a solid state drive.

[0008] In some embodiments, the processing circuit is configured to operate using a cache coherence protocol.

[0009] In some embodiments, the first data is stored in a first storage medium and includes at least one of an optimizer state, a gradient, and a weight calculated for a machine learning program.

[0010] In some embodiments, the second data is stored in the first storage medium and includes at least one of activations of the machine learning program, input batches, and checkpoint data.

[0011] In some embodiments, the first attribute comprises a state of computational logic used to train the machine learning program.

[0012] In some embodiments, the second attribute includes a data type.

[0013] In some embodiments, the first device is configured to: perform calculations based on the first data, generate third data based on the calculations, and transmit the third data, wherein the processing circuit is configured to: receive the third data; and store the third data in the first storage medium.

[0014] In some embodiments, the processing circuit is further configured to: receive a signal indicating a status of the first device; and update a parameter associated with the task.

[0015] In some embodiments, the state of the first device includes available memory of the first device, and the parameter includes a batch size of training data used to train the machine learning program.

[0016] One or more embodiments of the present disclosure also relate to a method, comprising: identifying first data associated with a task, the task being associated with a machine learning program; determining a first attribute associated with the first data; providing the first data to a first device based on determining the first attribute; identifying second data associated with the task, the task being associated with the machine learning program; determining a second attribute associated with the second data; and providing the second data to a second device different from the first device, wherein the machine learning program is configured to generate a result based on the input.

[0017] These and other features, aspects and advantages of the embodiments of the present disclosure will be more fully understood when considered with respect to the following detailed description, the appended claims and the accompanying drawings.Of course, the actual scope of the invention is defined by the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0019] Figure 1A conceptual layout diagram depicting a ML model undergoing an AI pipeline workload (e.g., machine learning (ML) training) according to one or more embodiments.

[0020] Figure 2 A block diagram depicts a computing system for building an ML model based on a workload of an AI pipeline in accordance with one or more embodiments.

[0021] Figure 3 A block diagram depicts an offload engine in accordance with one or more embodiments.

[0022] Figure 4 Depicted is a process for moving compute tensors between a host processor and a task processor in accordance with one or more embodiments.

[0023] Figure 5 A flow diagram of a process for moving tensor data by a host processor is depicted in accordance with one or more embodiments.

[0024] Figure 6 A flow diagram of a process for moving tensor data by a task processor is depicted in accordance with one or more embodiments. DETAILED DESCRIPTION

[0025] Hereinafter, example embodiments will be described in more detail with reference to the accompanying drawings, in which the same reference numerals always represent the same elements. However, the present disclosure may be implemented in various different forms and should not be construed as being limited to the embodiments shown herein. On the contrary, these embodiments are provided as examples so that the present disclosure will be thorough and complete, and will fully convey the aspects and features of the present disclosure to those skilled in the art. Therefore, unnecessary processing, elements and techniques for a person of ordinary skill in the art to fully understand the aspects and features of the present disclosure may not be described. Unless otherwise noted, throughout the accompanying drawings and written descriptions, the same reference numerals represent the same elements, and therefore, their descriptions may not be repeated. In addition, in the accompanying drawings, for clarity, the relative sizes of elements, layers and regions may be exaggerated and / or simplified.

[0026] Embodiments of the present disclosure are described below with reference to block diagrams and flow charts. Therefore, it should be understood that each block of the block diagram and flow chart can be realized in the form of a computer program product, an overall hardware embodiment, a combination of hardware and a computer program product, and / or an instruction, operation, step, and similar words (e.g., executable instructions, instructions for execution and / or program codes, etc.) on a computer-readable storage medium for execution, a system, a computing device, a computing entity, etc. For example, the retrieval, loading, and execution of the code can be performed sequentially so that an instruction is retrieved, loaded, and executed once. In some example embodiments, retrieval, loading, and / or execution can be performed in parallel so that multiple instructions are retrieved, loaded, and / or executed together. Therefore, such an embodiment can produce a machine that performs a specific configuration of the steps or operations specified in the block diagram and flow chart. Therefore, the block diagram and flow chart support various combinations of embodiments for executing specified instructions, operations, or steps.

[0027] The process of building a machine learning (ML) model (also referred to as a machine learning program) can be long and complex. An AI pipeline may identify a series of tasks or workloads that may need to be completed for building an ML model. For example, training data may need to be collected, the data may need to be preprocessed, features may need to be extracted from the data, and the model may need to be trained and validated using the extracted features. One or more operations for completing a task or workload of an AI pipeline may be performed serially and / or in parallel by one or more task processors.

[0028] Due to the large amount of data and data resources typically involved in the tasks (AI pipeline tasks), one or more of the AI ​​pipeline tasks (such as training) can be very computationally intensive and data intensive. For example, an ML model may include billions of model parameters including weights, activation functions, and gradient functions. One or more of the model parameters may need to be updated during an iteration of model training (e.g., via backpropagation). The number of iterations for updating the model parameters may depend on the batch size of examples considered by the model for training at each iteration.

[0029] In general, embodiments of the present disclosure relate to systems and methods for processing tasks in an AI pipeline using a heterogeneous memory system. Although various embodiments are described using ML training as a task, those skilled in the art will recognize that the embodiments can be extended to other tasks in an AI pipeline. A heterogeneous memory system can help solve the memory wall problem that can occur in the case of training large ML models, which also includes costs.

[0030] Despite the rapid improvements being made to memory and processing unit performance, the underlying memory technology may still not keep up with the growth of models (known as the "memory wall"). For example, even with the improvement of graphics processing unit (GPU) technology, a single GPU may not be sufficient to fit the parameters required for model training. Therefore, the computational and storage capacity of the GPU may limit the batch size used for training. The smaller the batch size, the longer it may take to train the model. Although multiple GPUs can be used to solve the memory wall problem, there may be a limit to the total number of GPUs that can be installed. In addition, using additional GPUs may increase the overall cost of the AI ​​system.

[0031] Using a heterogeneous memory system during ML training can provide a large working memory (e.g., an aggregated memory space), which can allow data to be selectively transferred between memory devices in the heterogeneous memory system to help solve, for example, memory wall problems. In some embodiments, the host processor collects information about devices available in the heterogeneous memory system. The available devices may include one or more GPUs, host memories, and / or storage devices (e.g., solid-state drives (SSDs)), etc. During ML training, the host processor may selectively transfer part of the model data to a task processor (e.g., a GPU) or a storage device (e.g., an SSD) according to one or more attributes of the model data. For example, a host processor (e.g., a central processing unit (CPU)) may maintain in its memory a specific type of model data of a task processor that is not involved in the current calculation (e.g., model weights, gradients, optimizer states of an optimizer algorithm). Other types of model data that are not frequently used for ML training (e.g., activations, batches of inputs, and / or checkpoints (or referred to as model checkpoints), etc.) may be offloaded to the SSD. When one or more calculations are performed to perform a task, the host processor may identify the data required for the calculations and, if the data is stored in the host memory, the host processor may transfer the data to the task processor for performing the calculations. For example, the data may be transferred along with the command to perform the calculations. Similarly, data that is deemed unused for performing ML training may be offloaded from the task processor to the host processor.

[0032] Offloading model data from a task processor can free up the task processor to process additional workload operations and / or data associated with the operation. For example, for ML training tasks, the task processor may be able to handle larger batch sizes for training ML models. Increased batch sizes can allow for accelerated training times and throughput improvements to help accelerate AI pipelines.

[0033] Figure 1A conceptual layout diagram depicting an ML model 100 (e.g., a deep neural network) subjected to an AI pipeline workload (e.g., machine learning (ML) training) according to one or more embodiments. The ML model may also be referred to as an ML program. The ML model 100 may include one or more nodes 102a to 102d (collectively referred to as 102) associated with one or more model parameters. The model parameters may include weights (e.g., W1 to W7), activation functions (referred to as activations) (e.g., A1 to A7), and / or gradients (e.g., G1 to G7), among others. During a forward pass of the training process, the activation of the node 102 may be calculated based on the input training data, weights, and activation functions. In some embodiments, the input training data includes one or more training samples or training examples. The number of training examples may be defined by an input batch size that may depend, for example, on the computational resources of a task processor available to perform the computations. The data set of training samples used to train the ML model may be divided into one or more batches of a given batch size. In some embodiments, before updating one or more model parameters, the samples in the input batch are fed to the ML model for prediction in a forward pass.

[0034] In some embodiments, one or more model parameters are updated during a backward pass (e.g., backpropagation) of the training process. An example parameter that is updated may be a weight of a node 102. In this regard, a loss may be calculated for the input training data via a loss function that compares a predicted output of the ML model 100 to an expected output (e.g., ground truth) 104. A gradient of the loss may be further calculated using an optimization function for the weight of the node 102, for example, during a backward pass. The weight may be adjusted based on the calculated gradient to minimize the loss.

[0035] Several iterations of forward and backward passes may be performed during the training of an ML model to generate a large amount of data that may need to be processed and stored. Model data (also referred to as model state) may be represented using a data structure such as a tensor (or referred to as tensor data). Therefore, hereinafter, model data generated during the training of an ML model will be referred to as a tensor or tensor data, but the present embodiment is not limited to a tensor.

[0036] Not all tensors may be actively used at a given time during the training process. For example, a tensor storing model parameters of one of the nodes (e.g., node 102b) may be actively used for computation, while a tensor storing model parameters of another of the nodes (e.g., node 102c) may be inactive.

[0037] Other types of model data (such as, for example, activations, training data sets, and / or model checkpoints) may also be used for model training, but not as frequently as model parameters. For example, a model checkpoint may represent the state of a model at a particular point in time (e.g., providing a model snapshot). In the event of an interruption in the model training process (e.g., a power outage), a model checkpoint may be required to allow model training to resume using the model checkpoint data once the interruption disappears.

[0038] It may be desirable to identify the various tensors generated for training an ML model and to offload tensors that are not used for current operations (hereinafter referred to as "unused tensors") from a task processor that handles computations to a first device, such as a device with low latency, high bandwidth, etc. (e.g., a host processor (e.g., CPU)). Some of the generated tensors may be tensors that are not typically used to perform computations (e.g., activations, checkpoints, etc.). It may be desirable to offload those tensors to a second device, such as a device with higher latency (e.g., SSD). The offloading of data may free up computational resources in the task processor to allow aspects of the training of the ML model to be adjusted. For example, the batch size of the input may be increased.

[0039] Figure 2 A block diagram depicts a computing system for building an ML model based on an AI pipeline workload according to one or more embodiments. In some embodiments, the computing system includes a host processor 200 that communicates with one or more task processors 202 and one or more storage devices (or data storage devices) 204 via an interface 206. Hereinafter, the host processor 200, the task processor 202, and the storage device 204 will be referred to as a "device". In some embodiments, the computing system may include Figure 2 At least one of the task processor 202, the host processor 200, the host memory 208, the interface 206, and the storage device 204 shown in FIG.

[0040] In some embodiments, interface 206 connects task processor 202 and storage device 204 to host processor 200 to allow the host processor to coherently access different types of memory in task processor 202 and storage device 204 as part of the host's memory space. In some embodiments, interface 206 includes a compute express link (CXL) switch configured with one or more CXL protocols (e.g., CXL.io, CXL.mem, and / or CXL.cache), but embodiments are not limited thereto. For example, in some embodiments, the interface 206 (e.g., the connector and its protocol) may include (or may conform to) a cache coherent interconnect for accelerators (CCIX), a dual in-line memory module (DIMM) interface, a small computer system interface (SCSI), non-volatile memory express (NVMe), a peripheral component interconnect express (PCIe), remote direct memory access (RDMA) over Ethernet, serial advanced technology attachment (SATA), Fibre Channel, serial attached SCSI (SAS), NVMe over fiber (NVMe-oF), iWARP protocol, InfiniBand protocol, 5G wireless protocol, Wi-Fi protocol, and / or Bluetooth protocol, etc. In various embodiments, the interface 206 (e.g., the connector and its protocol) includes various general-purpose interfaces (such as, for example, Ethernet and / or Universal Serial Bus (USB), etc.).

[0041] The host processor 200 may be a processing circuit (e.g., such as a general-purpose processor or central processing unit (CPU) core of a host computing device coupled to the host memory 208). The host memory 208 may be considered to be a high-performance main memory (e.g., primary memory) of the host device. For example, in some embodiments, the host memory 208 may include (or may be) a volatile memory (e.g., such as a dynamic random access memory (DRAM)). However, the present disclosure is not limited thereto, and the host memory 208 may include (or may be) any suitable high-performance main memory (e.g., primary memory) replacement for the host device as would be known to those skilled in the art. For example, in other embodiments, the host memory 208 may be a relatively high-performance non-volatile memory (e.g., NAND flash memory, phase change memory (PCM), resistive RAM, spin-transfer torque RAM (STTRAM), any suitable memory based on PCM technology and memristor technology, and / or resistive random access memory (ReRAM)), and may include, for example, chalcogenides, etc.

[0042] In some embodiments, the task processor 202 includes a GPU, but the embodiments are not limited thereto. For example, the task processor 202 may also take the form of a tensor processor, a coprocessor, or any other processing device configured to effectively process a task. In some embodiments, the task processor 202 may be a processing device for processing a task. For example, the task processor 202 may include at least one of a graphics processor, a tensor processor, and a coprocessor.

[0043] In some embodiments, the storage device 204 may be a storage device for storing data. For example, the storage device 204 may be an auxiliary memory device (such as, for example, an SSD). In some embodiments, the storage device 204 is implemented as a computing storage device (e.g., an SSD or a field programmable gate array (FPGA) with an embedded processor). However, the present disclosure is not limited thereto, and in other embodiments, the storage device 204 may include (or may be) any suitable storage device (such as, for example, a magnetic storage device (e.g., a hard disk drive (HDD)), etc.), an optical storage device (e.g., a Blu-ray drive, a compact disk (CD) drive, a digital versatile disk (DVD) drive, etc.) and / or other types of flash memory devices (e.g., a USB flash drive, etc.), etc.). In various embodiments, the storage device 204 may conform to a large form factor standard (e.g., a 3.5-inch hard disk drive form factor), a small form factor standard (e.g., a 2.5-inch hard disk drive form factor), an M.2 form factor, and / or an E1.S form factor, etc. In other embodiments, the storage device 204 may conform to any suitable or desired derivative of these form factors.

[0044] In some embodiments, the host processor 200 manages an AI pipeline including one or more tasks to be executed to build the ML model 210. For example, the host processor 200 may send commands to one or more of the task processors 202 for performing one or more tasks of the AI ​​pipeline and / or one or more calculations associated with the tasks. For example, the host processor 200 may send commands to the task processors 202 for training the ML model 210 based on a batch of training samples input. One or more task processors 202 may perform the tasks and / or calculations of the commands serially or in parallel. For example, the training samples may be split across two or more task processors in the task processors 202 for training the ML model in parallel.

[0045] In some embodiments, the host processor 200 includes an offload engine 212, which is configured to monitor operations or calculations (collectively referred to as calculations) associated with a task. For example, for an ML training task, the offload engine 212 may monitor the calculations associated with the nodes 102 of the ML model 210, the stages of the calculations (e.g., forward pass or backward pass), and / or the profiles of tensors associated with the calculations. The offload engine 212 may orchestrate the transfer or offloading of ML data (e.g., tensors) to one or more of the devices 202, 204 via the interface 206. The offload engine 212 may also detect the offloading of one or more unused tensors from the task processor 202 to the host processor 200.

[0046] For example, the task processor 202 may offload unused ML data (e.g., unused tensors) to the host memory 208. In some embodiments, the task processor identifies unused tensors for offloading. In some embodiments, the offload engine 212 identifies unused tensors. Whether a tensor is identified as unused may depend on one or more attributes of the tensor. Such attributes may be stored in a tensor profile and may include, for example, an access mode (or referred to as a tensor access mode) of the tensor (e.g., a memory block accessed by the tensor) and / or a timestamp associated with the tensor (e.g., the last time the tensor was used relative to the start of a training interval), etc. The task processor 202 and / or the offload engine 212 may review one or more of the attributes and offload the tensor to the host processor 200 based on one or more of the attributes. For example, if at least a threshold amount of time has passed since the tensor was last accessed based on the timestamp of the tensor, the task processor 202 and / or the offload engine 212 may offload the tensor.

[0047] In some embodiments, the tensor is transferred by the task processor 202 via an unload (or store) command. The unload command may identify, for example, the tensor to be unloaded (e.g., tensor ID) and the destination of the unload. In some embodiments, the destination of the unload may be a memory location in the host memory 208. The transfer may be implemented, for example, via direct memory access to the host memory 208 using the interface 206.

[0048] The offload engine 212 can monitor the location of tensors (e.g., a device storing tensors) and the status (e.g., progress) of the computational logic (e.g., computational operations) for ML model training. The offload engine 212 can determine that tensors required for the upcoming computational operations have been offloaded to the host memory 208, and provide the tensors to the task processor 202 for performing the computational operations. For example, the offload engine 212 can execute (e.g., via a storage command) to write the tensors to the memory of the task processor 202.

[0049] In some embodiments, the host processor 200 retains a portion of the unloaded tensor data in the host memory 208 while further offloading a portion of other tensor data to the storage device 204. The selection of tensor data to be retained in the host memory 208 and the tensor data to be offloaded to the storage device 204 may depend on, for example, the data type of the ML data. For example, specific model parameters (such as optimizer states, gradients, and model weights) that are typically used to perform calculations during a forward pass or a backward pass of a training operation may be retained in the host memory 208. Other model data (such as activations, input batches, and checkpoints) that may not be frequently accessed during model training (if any) may be offloaded to the storage device 204.

[0050] Figure 3 A block diagram depicts an offload engine 212 according to one or more embodiments. Offload engine 212 may include a controller 300 coupled to a cache 302. In some embodiments, controller 300 takes the form of a CXL controller that complies with the CXL.mem protocol, the CXL.io protocol, and / or the CXL.cache protocol. In some embodiments, offload engine 212 may include Figure 3 At least one of the controller 300, cache 302, and tensor map 310 shown in FIG.

[0051] The controller 300 may include at least one processing component embedded thereon (such as, for example, an ARM processor, a graphics processing unit (GPU), a field programmable gate array (FPGA), and / or another digital circuit (e.g., a microcontroller, a microprocessor, a digital signal processor, or a logic device (e.g., an application specific integrated circuit (ASIC))). The processing component may help implement the functions described herein. In this regard, in some embodiments, the controller 300 is configured to collect information of the devices 200, 202, 204 in the heterogeneous memory system. For example, the controller 300 may collect capacity information and bandwidth information of the devices.

[0052] In some embodiments, controller 300 monitors the computational logic of AI pipeline tasks and orchestrates the transfer of data from one device to another device and / or the sharing of data (e.g., between host memory 208, task processors 202, and / or storage devices 204) in a consistent manner. In some embodiments, CXL controller 300 includes a device consistency agent 304 for ensuring the consistency of cache 302 with other caches in a heterogeneous memory system. In some embodiments, similar device consistency agents reside in task processors 202 and / or storage devices 204 to ensure the consistency of their internal caches.

[0053] In some embodiments, cache 302 stores at least a portion of the ML data used by the AI ​​pipeline tasks. For ML training tasks, cache 302 may store computational logic and associated data (hereinafter referred to as computational tensors) 306. The cache may also store all or part (if any) of other ML data 308 (e.g., activations, batches of inputs, and / or checkpoints) that may be accessed infrequently after training begins. In some embodiments, the cache is flushed after the data is offloaded to task processor 202, host memory 208, and / or storage device 204. ML data 308 may also be included in tensors.

[0054] Compute tensors 306 and other ML data 308 may have different attributes associated with them (e.g., in a tensor configuration file). Attributes may include, for example, a tensor identifier (ID), an access timestamp (relative to the time of the start of an iteration), a tensor access mode (e.g., a memory block accessed by a tensor), a tensor access interval (e.g., a starting memory address and an ending memory address in the device being accessed), etc. In some embodiments, the tensor configuration file also indicates the data type of the tensor. The data type may indicate whether the tensor is one of the compute tensors 306 (e.g., optimizer states, gradients, model weights, or other computational parameters) or other ML data 308 (e.g., activations, batches of inputs, checkpoint data, etc.). The tensor configuration file may accompany the tensor when the tensor is moved from one device to another. In some embodiments, the tensor configuration file is part of the tensor data.

[0055] The tensor profile may be used by the offload engine 212 and / or the task processor 202 to determine whether a tensor should be offloaded and to which device. For example, the controller 300 may monitor the computation logic and provide the computation tensors 306 (e.g., weight tensors) stored in the cache 302 to one or more task processors 202 (e.g., to the memory of one or more task processors) for performing computations during ML model training. In some embodiments, other ML data 308 stored in the cache 302 that may not be needed (or not often needed) to perform computations during the training process may be offloaded to the storage device 204.

[0056] In some embodiments, the cache 302 also stores a tensor map 310. The tensor map 310 may be used to identify, for one or more tensor IDs, an identifier of a device (e.g., a host processor 200, a task processor 202, or a storage device 204) that currently stores the tensor. In some embodiments, the tensor map 310 includes the data type of the tensor (e.g., a compute tensor or other ML data tensor). The tensor map 310 may allow for identification of compute tensors stored in the cache 302 or host memory 208 that may be needed for computation, thereby being moved to the memory of the task processor 202 to perform the computation. The move may be via a write command issued by the controller 300. The target address of the write command may be the address of the task processor 202 that will perform the computation. The writing of data may be, for example, via direct memory access to the target address using the interface 206.

[0057] The tensor graph 310 may also allow identification of other ML data that may not be needed for computation and thus be moved to the storage device 204. This may allow the host memory 208 to be freed up, which in turn may allow data that may not be needed by the task processor 202 to perform the current computation to be unloaded to the host memory 208 (e.g., via the host processor 200). Freeing up memory in the task processor 202 may help speed up training and help improve throughput. In this regard, the task processor 202 may send a signal indicating the status of the task processor 202 (e.g., available memory). Based on the signal, the host processor 200 may update parameters associated with the ML training task (e.g., the batch size of the training data used to train the ML model). For example, the batch size of the input of the training data may be increased. The increase in the batch size of the input may reduce the number of iterations required to train the ML model and help speed up the time it takes to train the ML model.

[0058] Figure 4 Depicted is a process for moving computational tensors between a host processor 200 and a task processor 202 according to one or more embodiments. During the start of a training process, ML data 400 (e.g., an input training batch) stored in a storage device 204 is provided to one or more task processors 202 for performing one or more computations. For example, the ML data 400 may be loaded into a cache 302 of a host processor 200 and then provided to the task processor 202 (e.g., in a cache coherent manner). The computations performed by the task processor 202 may include, for example, computing weights, activations, and gradients of one or more nodes 102 during an iteration (e.g., step i) 402 of training (e.g., an i-th forward pass and backward pass). One or more computational tensors of a first state 404 may be generated during the i-th iteration 402 of training. The computational tensors may be associated with Figure 3The computation tensor 306 of can be similar to that of . The computation tensor can be updated during the i-th iteration of training based on the updated model parameters to generate a computation tensor of the second state 410 .

[0059] In some embodiments, one or more of the computed tensors may be offloaded by the task processor 202 to the host processor 200. For example, one or more gradient tensors 406 may be offloaded to the host processor 200 during a backward pass of training. For example, once the gradient tensor 406 has been computed, the gradient tensor 406 may be offloaded by the task processor 202. The offloaded gradient tensor 406 may be stored, for example, in the cache 302 or the host memory 208. In some embodiments, the tensor map is updated to indicate that the gradient tensor 406 is stored in the host memory 108.

[0060] In some embodiments, the task processor 202 may perform an adjustment of one or more model parameters (e.g., weights) of one or more nodes 102 based on the calculated gradient tensor 406 to generate an updated parameter tensor 408. The updated parameter tensor 408 may also be offloaded to the host processor 200 and stored in the cache 302 and / or the host memory 208. The updated parameter tensor 408 may include Figure 3 One or more of the computational tensors 306 .

[0061] In some embodiments, the controller 300 determines, based on monitoring the computation logic, that the updated parameter tensor 408 is to be used to generate the computation tensor of the second state 410. The controller 300 may transmit the updated parameter tensor 408 to the task processor 202 based on the determination.

[0062] Although not in Figure 4 2 , but it should be understood that the task processor 202 and / or the host processor 200 may utilize and / or store other ML data 308 (such as activations, checkpoints, etc.). In some embodiments, the controller 300 offloads such other ML data 308 to the storage device 204.

[0063] Figure 5 A flow chart is depicted of a process for moving tensor data by the host processor 200 in accordance with one or more embodiments. The process may be performed, for example, by the offload engine 212 during an iteration of model training.

[0064] Processing begins, and in act 500, the offload engine 212 identifies first data associated with a task associated with (e.g., for building) an ML model. The task may be, for example, an ML training operation. The first data may be, for example, a computed tensor 306 stored in cache 302 or host memory 208. The built ML model may be used to generate predictions based on received inputs.

[0065] In act 502, the offload engine 212 determines a first attribute associated with the first data. The first attribute may include a state of a computation logic used to train the ML model. The state of the computation logic may be used to identify a computation to be performed for the training of the ML model, which may require the first data to perform the computation. The computation may be a computation during a forward pass or a backward pass of the training.

[0066] In act 504, the first data is provided to the first device based on determining the first attribute. The first device may be the task processor 202. The transfer of the data may require sending a storage command to the task processor 202. The task processor 202 may perform a calculation using the first data. The calculation may generate additional data (e.g., third data), which may be retained in the task processor 202 or offloaded to the host processor 200 to free up memory of the task processor.

[0067] In action 506, the offload engine 212 identifies second data associated with the task associated with (e.g., for building) the ML model. The second data may be, for example, one of the other ML data 308 stored in the cache 302 or the host memory 208 that may not be needed (or not often needed) to perform the computation.

[0068] In action 508, the offload engine 212 determines a second attribute associated with the second data. The second attribute may include a data type (eg, a data type of the second data). The data type may be at least one of activation data, input data, and / or checkpoint data, among others.

[0069] In act 510, offload engine 212 provides the second data to a second device different from the first device (eg, based on the second attribute).

[0070] Figure 6 A flow chart is depicted of a process for moving tensor data by the task processor 202 according to one or more embodiments. The process may be performed, for example, during an iteration of model training.

[0071] The process begins, and in act 600, the task processor 202 identifies a tensor. The identified tensor may be a tensor accessed by the task processor 202 during a computation and / or a tensor generated by the task processor as a result of a computation, etc.

[0072] In action 602, the task processor 202 determines whether to offload the tensor to the host processor 200. In this regard, the task processor 202 (or the offload engine 212) may review one or more attributes of the tensor (e.g., stored in a tensor configuration file) to determine whether the tensor can be considered unused. The reviewed attributes may include, for example, the access pattern of the tensor (e.g., the memory block accessed by the tensor) and / or the timestamp associated with the tensor (e.g., the last time the tensor was used relative to the start of the training interval), etc. For example, the task processor 202 (or the offload engine 212) may determine that the tensor is unused and therefore the tensor should be offloaded if at least a threshold amount of time has passed since the tensor was last accessed based on the timestamp of the tensor.

[0073] In another example, an offload decision may be made based on dynamic tensor access patterns (e.g., random, sequential, vectorized, etc.) that are tracked during runtime. For example, if the tensor access pattern during a training iteration is regular, one or more of the tensors may be offloaded to optimize the performance of the underlying hardware architecture. For example, some tensors may be offloaded to the GPU / task processor 202 to optimize the GPU for parallel computing, and some tensors may be offloaded to the CPU / host processor 200 to optimize the CPU for single-threaded performance. In some embodiments, the task processor 202 (or offload engine 212) also considers available memory and other computing resources for determining whether to offload tensors.

[0074] If the tensor is not to be unloaded, the tensor is maintained in the memory of the task processor 202 .

[0075] One or more embodiments of the present disclosure may be implemented with one or more processors (also referred to as processing circuits). The term processor may refer to one or more processors and / or one or more processing cores. One or more processors may be hosted in a single device or distributed on multiple devices (e.g., on a cloud system). The processor may include, for example, an application-specific integrated circuit (ASIC), a general or dedicated central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), and a programmable logic device (such as a field programmable gate array (FPGA)). In a processor, as used herein, each function is performed by hardware configured (i.e., hard-wired) to perform the function, or by more general hardware (such as a CPU) configured to execute instructions stored in a non-temporary storage medium (e.g., a memory). The processor may be manufactured on a single printed circuit board (PCB) or distributed on a number of interconnected PCBs. The processor may include other processing circuits; for example, the processing circuit may include two processing circuits (FPGA and CPU) interconnected on a PCB.

[0076] It will be understood that although the terms "first", "second", "third", etc. may be used herein to describe various elements, components, regions, layers and / or parts, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer or part from another element, component, region, layer or part. Therefore, without departing from the spirit and scope of the inventive concept, the first element, first component, first region, first layer or first part discussed herein may be referred to as the second element, second component, second region, second layer or second part.

[0077] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the inventive concept. In addition, unless explicitly stated, the embodiments described herein are not mutually exclusive. The aspects of the embodiments described herein may be combined in some embodiments.

[0078] As used herein, unless the context clearly indicates otherwise, the singular form is intended to also include the plural form. It will also be understood that the terms "including" and / or "comprising" when used in this specification specify the existence of stated features, integers, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Statements such as "at least one of..." modify the entire column of elements when following a column of elements, rather than modifying the individual elements in the column. In addition, the use of "may" when describing the embodiments of the inventive concept represents "one or more embodiments of the present disclosure". In addition, the term "exemplary" is intended to represent an example or description. As used herein, the terms "use", "being used" and "being used" may be considered to be synonymous with the terms "utilization", "being utilized" and "being utilized", respectively.

[0079] Although exemplary embodiments of systems and methods for processing tasks of an AI pipeline have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. Therefore, it will be understood that systems and methods for processing tasks of an AI pipeline constructed according to the principles of the present disclosure may be implemented differently than as specifically described herein. The disclosure is further defined in the appended claims and their equivalents.

[0080] Systems and methods for processing storage transactions may include one or more combinations of the features set forth in the following claims.

[0081] Statement 1. A system comprising: a first storage medium; and a processing circuit that communicates with the first storage medium, the processing circuit being configured to: identify first data associated with a task, the task being associated with a machine learning program; determine a first attribute associated with the first data; provide the first data to a first device based on determining the first attribute; identify second data associated with the task, the task being associated with the machine learning program; determine a second attribute associated with the second data; and provide the second data to a second device different from the first device, wherein the machine learning program is configured to generate a result based on an input.

[0082] Statement 2. The system of statement 1 wherein the first device comprises at least one of a graphics processor, a tensor processor, and a coprocessor.

[0083] Statement 3. The system of statement 2 wherein the second device comprises a solid state drive.

[0084] Statement 4. The system of statement 1 wherein the processing circuit is configured to operate using a cache coherence protocol.

[0085] Statement 5. The system of statement 1, wherein the first data is stored in a first storage medium and includes at least one of an optimizer state, a gradient, and a weight calculated for a machine learning program.

[0086] Statement 6. The system of statement 1, wherein the second data is stored in the first storage medium and includes at least one of activations of the machine learning program, input batches, and checkpoint data.

[0087] Statement 7. The system of statement 1, wherein the first attribute comprises a state of computational logic used to train a machine learning program.

[0088] Statement 8. The system of statement 1 wherein the second attribute comprises a data type.

[0089] Statement 9. The system of statement 1, wherein the first device is configured to: perform calculations based on the first data, generate third data based on the calculations, and transmit the third data, wherein the processing circuit is configured to: receive the third data; and store the third data in the first storage medium.

[0090] Statement 10. The system of statement 1 wherein the processing circuit is further configured to: receive a signal indicative of a status of the first device; and update a parameter associated with the task.

[0091] Statement 11. The system of statement 10, wherein the state of the first device comprises available memory of the first device, and the parameter comprises a batch size of training data used to train the machine learning program.

[0092] Statement 12. A method comprising: identifying first data associated with a task, the task being associated with a machine learning program; determining a first attribute associated with the first data; providing the first data to a first device based on determining the first attribute; identifying second data associated with the task, the task being associated with the machine learning program; determining a second attribute associated with the second data; and providing the second data to a second device different from the first device, wherein the machine learning program is configured to generate a result based on the input.

[0093] Statement 13. The method of statement 12 wherein the first device comprises at least one of a graphics processor, a tensor processor, and a coprocessor.

[0094] Statement 14. The method of statement 13 wherein the second device comprises a solid state drive.

[0095] Statement 15. The method of statement 12, wherein the first data is stored in a storage medium and includes at least one of an optimizer state, a gradient, and a weight calculated for a machine learning program.

[0096] Statement 16. The method of statement 12, wherein the second data is stored in a storage medium and includes at least one of activations of the machine learning program, input batches, and checkpoint data.

[0097] Statement 17. The method of statement 12, wherein the first attribute comprises a state of computational logic used to train a machine learning program.

[0098] Statement 18. The method of statement 12 wherein the second attribute comprises a data type.

[0099] Statement 19. The method of statement 12, wherein the first device performs calculations based on the first data, generates third data based on the calculations, and transmits the third data, wherein the method further comprises: receiving the third data; and storing the third data in a storage medium.

[0100] Statement 20. The method of statement 12 further comprising: receiving a signal indicative of a status of the first device; and updating a parameter associated with the task.

Claims

1. A system for processing a task, comprising: a first storage medium; as well as A processing circuit communicates with the first storage medium, the processing circuit being configured to: identifying first data associated with the task, the task being associated with a machine learning program; determining a first attribute associated with the first data; providing first data to the first device based on determining the first attribute; identifying second data associated with the task; determining a second attribute associated with the second data; and The second data is provided to a second device different from the first device, wherein the machine learning program is configured to generate a result based on the input.

2. The system of claim 1, wherein: The first device includes at least one of a graphics processor, a tensor processor, and a coprocessor.

3. The system of claim 2, wherein: The second device includes a solid state drive.

4. The system of claim 1, wherein: The processing circuitry is configured to operate using a cache coherence protocol.

5. The system of claim 1, wherein: The first data is stored in a first storage medium and includes at least one of an optimizer state, a gradient, and a weight calculated for a machine learning program.

6. The system of claim 1, wherein: The second data is stored in the first storage medium and includes at least one of activations of the machine learning program, input batches, and checkpoint data.

7. The system of claim 1, wherein: The first attribute includes the state of the computational logic used to train the machine learning program.

8. The system of claim 1, wherein: The second attribute includes a data type of the second data.

9. The system of claim 1, wherein: The first device is configured to: perform calculation based on the first data to generate third data, and transmit the third data, and The processing circuit is further configured to: receive third data; and store the third data in the first storage medium.

10. The system of any one of claims 1 to 9, wherein: The processing circuit is further configured to: receiving a signal indicating a status of the first device; and Parameters associated with the task are updated.

11. The system of claim 10, wherein: The state of the first device includes the capacity of available memory of the first device, and the parameter includes a batch size of training data used to train the machine learning program.

12. A method for processing a task, comprising: identifying first data associated with the task, the task being associated with a machine learning program; determining a first attribute associated with the first data; providing first data to the first device based on determining the first attribute; identifying second data associated with the task; determining a second attribute associated with the second data; as well as The second data is provided to a second device different from the first device, wherein the machine learning program is configured to generate a result based on the input.

13. The method of claim 12, wherein: The first device includes at least one of a graphics processor, a tensor processor, and a coprocessor.

14. The method of claim 13, wherein: The second device includes a solid state drive.

15. The method of claim 12, wherein: The first data is stored in a storage medium and includes at least one of an optimizer state, a gradient, and a weight calculated for a machine learning program.

16. The method of claim 12, wherein: The second data is stored in a storage medium and includes at least one of activations of the machine learning program, input batches, and checkpoint data.

17. The method of claim 12, wherein: The first attribute includes the state of the computational logic used to train the machine learning program.

18. The method of claim 12, wherein: The second attribute includes a data type of the second data.

19. The method of claim 12, wherein: The first device performs calculation based on the first data to generate third data, and transmits the third data, and The method further includes: receiving third data; and storing the third data in a storage medium.

20. The method of any one of claims 12 to 19, further comprising: receiving a signal indicating a status of the first device; as well as Parameters associated with the task are updated.

Images