Method for model training, host and storage device

Abstract

The invention provides a method for model training, a host and a storage device. The method includes instructing, by a host device, a graphics processing unit (GPU) to load first intermediate data of a current layer of the model from a dynamic random access memory (DRAM) of a storage device into a memory of the GPU, where the current layer is a layer of the model that is performing computations; and based on the space capacity of the DRAM, notifying, by the host device, the storage device to pre-fetch second intermediate data of the layer to be calculated of the model from the NAND of the storage device to the DRAM of the storage device, the first intermediate data for the current layer is used by the GPU in back propagation of the model to perform computations for the current layer.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a method, a host, and a storage device for model training. Background Art

[0002] With the development of artificial intelligence (AI) technology, models are becoming larger and larger, and the demand for GPU memory is increasing. In order to obtain larger models with higher model accuracy and faster network convergence speed, training requires deeper network depth and larger training samples (batch size), which may be limited by the memory capacity associated with the GPU, which is commonly known as the "memory wall" problem. Summary of the invention

[0003] The present disclosure provides a method, a host, and a storage device for model training to solve part or all of the above problems.

[0004] According to one aspect of the present disclosure, a method for training a neural network model is provided. The method may include: a host device instructing a graphics processor GPU to load the first intermediate data of the current layer of the model from a dynamic random access memory DRAM of a storage device into the memory of the GPU, wherein the current layer is the layer of the model that is performing calculations; and based on the space capacity of the DRAM, the host device notifies the storage device to pre-fetch the second intermediate data of the layer to be calculated of the model from the NAND of the storage device to the DRAM of the storage device, wherein the first intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer.

[0005] In an embodiment, pre-fetching of the second intermediate data is performed at least partially in parallel with calculation for the current layer.

[0006] In an embodiment, notifying the storage device to prefetch the second intermediate data may include: traversing the layer to be calculated of the model as a prefetch layer, and based on the fact that the DRAM has remaining space and prefetching of the second intermediate data has not been performed for the prefetch layer, notifying the storage device to prefetch the second intermediate data for the prefetch layer.

[0007] In an embodiment, the method may further include: in the forward propagation of the model, instructing the GPU to unload third intermediate data generated by calculation of the layer of the model from the memory of the GPU to the NAND of the storage device.

[0008] In an embodiment, the method may further include: before training the model, based on the memory amount of the GPU and the execution time of the forward propagation, obtaining information on unloading the third intermediate data to one or more layers of the NAND in multiple layers of the model; wherein, in the forward propagation of the model, instructing the GPU to unload the third intermediate data may include: based on the information, instructing the GPU to unload the third intermediate data in the forward propagation of the model.

[0009] In an embodiment, obtaining information of unloading the third intermediate data to one or more layers of the model in multiple layers to the NAND based on the memory amount of the GPU and the execution time of the forward propagation may include: deriving the information based on the total amount of remaining intermediate data being less than or equal to the memory amount of the GPU and the total unloading time being less than or equal to the execution time of the forward propagation, wherein the total amount of remaining intermediate data is based on the amount of fourth intermediate data of the layers in the multiple layers of the model other than the layer in which the third intermediate data is unloaded to the NAND, and wherein the total unloading time is based on the unloading time of the layer in which the third intermediate data is unloaded to the NAND.

[0010] In an embodiment, the first intermediate data comprises activation values, and the calculation for the current layer comprises gradient calculation with respect to the activation values.

[0011] According to one aspect of the present disclosure, a method for training a neural network model is provided, the method being applied to a storage device including a dynamic random access memory DRAM and a NAND, the method comprising: receiving, by the storage device, a notification of performing a data prefetch operation from a host, wherein the storage device comprises: a dynamic random access memory DRAM and a NAND, and based on the received notification, prefetching, by the storage device, intermediate data of a layer to be calculated of the model from the NAND to the DRAM.

[0012] In an embodiment, the method further includes: sending intermediate data of the current layer of the model to the memory of a graphics processor GPU, wherein the current layer of the model is the layer of the model that is performing calculations, wherein the intermediate data of the current layer is stored in the DRAM, and wherein the intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer.

[0013] In an embodiment, pre-fetching of intermediate data of a layer to be calculated of the model is performed at least partially in parallel with the calculation for the current layer.

[0014] In an embodiment, the intermediate data of the layer to be calculated of the model is first intermediate data, and wherein the method further comprises: in the forward propagation of the model, receiving second intermediate data, the second intermediate data being data generated by the calculation of the layer of the model and unloaded from the memory of the GPU to be stored in the NAND.

[0015] In an embodiment, sending the intermediate data of the current layer includes: based on the Compute Quick Link CXL protocol, sending the intermediate data of the current layer stored in the DRAM to the memory of the GPU, wherein receiving the second intermediate data includes: based on the CXL protocol, receiving the second intermediate data generated by the calculation and unloaded from the memory of the GPU to be stored in the NAND.

[0016] In an embodiment, the storage device further comprises a memory-semantic solid state drive MS SSD.

[0017] According to one aspect of the present disclosure, a host is provided, comprising: a memory storing instructions; and a processor configured to execute the instructions to perform the following operations: instruct a graphics processor GPU to load first intermediate data of a current layer of a model from a dynamic random access memory DRAM of a storage device into a memory of the GPU, wherein the current layer is a layer of the model that is performing calculations; and based on the space capacity of the DRAM, notify the storage device to pre-fetch second intermediate data of a layer to be calculated of the model from a NAND of the storage device into the DRAM, wherein the first intermediate data of the current layer is used by the GPU in back propagation of the model to perform calculations for the current layer.

[0018] In an embodiment, pre-fetching of the second intermediate data is performed at least partially in parallel with calculation for the current layer.

[0019] In an embodiment, the processor is also configured to execute the instructions to perform the following operations: traverse the layer to be calculated of the model as a prefetch layer, and based on the fact that the DRAM has remaining space and prefetching of the second intermediate data has not been performed for the prefetch layer, notify the storage device to prefetch the second intermediate data for the prefetch layer.

[0020] In an embodiment, the processor is further configured to execute the instructions to perform the following operations: in the forward propagation of the model, instruct the GPU to unload third intermediate data generated by calculations for a layer of the model from the memory of the GPU to the NAND.

[0021] In an embodiment, the processor is also configured to execute the instructions to perform the following operations: before training the model, based on the amount of memory of the GPU and the execution time of the forward propagation, obtain information on unloading the third intermediate data to one or more layers of the NAND in multiple layers of the model; based on the information, instruct the GPU to unload the third intermediate data in the forward propagation of the model.

[0022] In an embodiment, the processor is further configured to execute the instructions to perform the following operations: derive the information based on the total amount of remaining intermediate data being less than or equal to the amount of memory of the GPU and the total unloading time being less than or equal to the execution time of the forward propagation, wherein the total amount of remaining intermediate data is based on the amount of fourth intermediate data of a layer among the multiple layers of the model excluding the layer in which the third intermediate data is unloaded to the NAND, and wherein the total unloading time is based on the unloading time of the layer in which the third intermediate data is unloaded to the NAND.

[0023] In an embodiment, the first intermediate data comprises activation values, and the calculation for the current layer comprises gradient calculation with respect to the activation values.

[0024] According to one aspect of the present disclosure, a storage device is provided, comprising: a dynamic random access memory DRAM and a NAND, wherein the storage device is configured to: receive a notification of performing a data prefetch operation from a host, and based on the received notification, prefetch intermediate data of a layer to be calculated of a model from the NAND to the DRAM.

[0025] In an embodiment, the storage device is also configured to: send intermediate data of the current layer of the model to the memory of a graphics processor GPU, wherein the current layer of the model is the layer of the model that is performing calculations, wherein the intermediate data of the current layer is stored in the DRAM, and wherein the intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer.

[0026] In an embodiment, pre-fetching of intermediate data of a layer to be calculated of the model is performed at least partially in parallel with the calculation for the current layer.

[0027] In an embodiment, the intermediate data of the layer to be calculated of the model is first intermediate data, and the storage device is also configured to: in the forward propagation of the model, receive second intermediate data, which is generated by the calculation of the layer of the model and is unloaded from the memory of the GPU to be stored in the NAND.

[0028] In an embodiment, the storage device is further configured to: send the intermediate data of the current layer stored in the DRAM to the memory of the GPU based on a Compute Quick Link (CXL) protocol, and receive the second intermediate data generated by the calculation that is unloaded from the memory of the GPU to be stored in the NAND based on the CXL protocol.

[0029] In an embodiment, the storage device further comprises a memory-semantic solid state drive MS SSD.

[0030] According to one aspect of the present disclosure, a system to which a storage device is applied is provided, the system comprising: a main processor; a memory; and the storage device, wherein the storage device is configured to execute a method for training a neural network model.

[0031] According to one aspect of the present disclosure, a host storage system is provided, comprising: a host; and a storage device, wherein the storage device is configured to execute a method for training a neural network model.

[0032] According to one aspect of the present disclosure, there is provided a data center system, comprising: a plurality of application servers; and a plurality of storage servers, wherein each storage server comprises a storage device, wherein the storage device is configured to execute a method for training a neural network model.

[0033] According to one aspect of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, a method for training a neural network model is implemented.

[0034] The technical solution provided according to the embodiments of the present disclosure brings at least the following beneficial effects: data offloading between the GPU and the storage device is used to solve the "memory wall" problem of the GPU, and data prefetching is used to speed up the training of the model, thereby achieving better training throughput; because the GPU and the storage device transmit data through direct communication, less CPU resources and main memory bandwidth can be used and robust performance can be provided; and, while the software implementation complexity is low, higher storage capacity and lower cost are provided for large model training.

[0035] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0037] Figure 1 Schematic diagram showing deep neural network (DNN) training iterations and data reuse patterns.

[0038] Figure 2 A graph showing the growth trend of the number of parameters of the SOTA model.

[0039] Figure 3 A schematic diagram of a distributed graphics processing unit (GPU) server training system is shown.

[0040] Figure 4 A system diagram showing offloading tensors to central processing unit (CPU) main memory.

[0041] Figure 5 System diagram showing offloading tensors to SSD.

[0042] Figure 6 A system diagram for model training according to an embodiment of the present disclosure is shown.

[0043] Figure 7 A diagram showing the interaction between the internal pre-fetching of a memory-semantic solid-state drive (MS SSD) and the host side according to an embodiment of the present disclosure.

[0044] Figure 8 A flowchart of a process for model training applied to a host according to an embodiment of the present disclosure is shown.

[0045] Fig. 9 A diagram showing a comparison between non-volatile memory express (NVMe) as backup storage and MS SSD as backup storage according to an embodiment of the present disclosure.

[0046] Fig.10 A flow chart showing a process of pre-fetching data from MS SSD NAND to DRAM in reverse propagation according to an embodiment of the present disclosure.

[0047] Fig.11 A diagram illustrating a process of pre-fetching unloaded intermediate data in reverse propagation according to an embodiment of the present disclosure.

[0048] Fig.12 A diagram illustrating an offloading process using a MS SSD as a backup storage according to an embodiment of the present disclosure.

[0049] Fig.13 The flowchart of the process of deriving the optimal unloading scheduling according to an embodiment of the present disclosure is shown.

[0050] Fig.14 A flowchart of a process for model training applied to a storage device according to an embodiment of the present disclosure is shown.

[0051] Fig.15A diagram showing a host according to an embodiment of the present disclosure.

[0052] Fig.16 A diagram illustrating a storage device according to an embodiment of the present disclosure.

[0053] Fig.17 is a diagram of a system using a storage device according to an embodiment of the present disclosure.

[0054] Fig.18 is a block diagram of a host storage system according to an embodiment of the present disclosure.

[0055] Fig.19 is a diagram of a data center to which a storage device is applied according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0058] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0059] Artificial intelligence models, for example, Deep Neural Networks (DNN) models consist of many interconnected layers through which samples are propagated. Figure 1 Schematic diagram showing DNN training iteration and data reuse mode. Figure 1As shown in the figure, an iterative calculation propagation of the model consists of two processes: forward propagation and back propagation. Forward propagation calculates the final output of the model based on the input and the activation generated in the middle. Back propagation calculates the gradient according to the activation value and the loss value. When all layers have completed the calculation, the weights of each layer are updated according to the weight gradient to reduce the error rate of the model output. Figure 1 , the intermediate data generated by the layer calculations for the model in the forward propagation (for example, activation A [2] ), can be stored, and in the back propagation, the stored intermediate data can be used for calculation (i.e., reused), for example, to calculate dA [2] .

[0060] In the following example embodiments, the intermediate data (or intermediate result) is taken as an activation (also referred to as an activation tensor, an activation value), and in the back propagation, the calculation for the current layer using the intermediate data is described by taking the gradient calculation as an example, but it should be understood that the present disclosure is not limited to this. For example, the intermediate data may also be other data generated by the calculation for the layer of the model in the forward propagation, and in the back propagation, the calculation for the current layer using the intermediate data may be any calculation related to the intermediate data.

[0061] Figure 2 The growth trend of the number of parameters of the SOTA (State Of The Art) model is shown. The large Transformer model has parameters that grow at an exponential rate of 240 times every two years. However, the memory of a single GPU only doubles every two years, and the growth of GPU memory cannot meet the growing memory requirements of AI model training. In order to obtain large models with higher model accuracy and faster network convergence speed, training requires deeper network depth and larger training samples, but GPUs cannot meet the memory requirements under this trend, which is commonly known as the "memory wall" problem.

[0062] Related technologies try to solve the "memory wall" problem in the following three ways:

[0063] 1) Use distributed GPU servers for training. Figure 3 A diagram showing a distributed GPU server training system is shown, Figure 3 As shown, multiple GPUs are used as parameter servers (PS) to expand memory, and a series of parallel methods (such as data parallelism, model parallelism, pipeline parallelism, etc.) are used to accelerate training.

[0064] 2) Offload the training model state or activate it to the CPU memory (CPU memory). Figure 4 A system diagram showing offloading tensors to CPU main memory. Figure 4 As shown, offloading involves buffering part of the tensors (i.e., the state or activation of the model) from the GPU's memory (GPU memory) to the CPU memory to avoid GPU memory overflow.

[0065] 3) Offload activation to a non-volatile memory express solid-state drive (NVMe SSD). Figure 5 A diagram showing offloading activation to an SSD. Figure 5 As shown in Figure 1, a traditional SSD (Solid State Drive) is used as a backup storage to buffer some activations in the GPU memory. Data is directly transferred between the GPU and the SSD through a point-to-point direct storage access method, which can reduce CPU resources and main memory usage.

[0066] However, there are problems with each attempt. For training using distributed GPU servers, storage capacity, training cost, and offloading solutions are not good options in addition to high GPU bandwidth. For offloading training model state or activation to CPU memory, in addition to the simplicity of the offloading solution, training cost, storage capacity, and robustness as training performance are not good options. Finally, for offloading activation to NVMe SSD, although the total cost of ownership (TCO) is small, the IO bandwidth of traditional SSD is a bottleneck. The advantages and disadvantages of the above three solutions are shown in Table 1:

[0067] Table 1

[0068]

[0069] In order to solve the "memory wall" problem of GPU and overcome the shortcomings of the above solutions, the present disclosure proposes a method for model training, which offloads the intermediate data (e.g., activation) generated during the training of the model (e.g., DNN) to a storage device (e.g., memory-semantic solid-state drive MS SSD), and loads the intermediate data faster through a pre-fetch algorithm to accelerate the training of large models. Figures 6 to 19 The method, host, and storage device for model training according to the present disclosure are specifically described.

[0070] The model here can be a deep neural network (DNN) model, or other models including multiple layers, for example, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a generative adversarial network (GAN) model, a long short-term memory network (LSTM) model, a residual network (ResNet, such as ResNet-1922) model, an attention mechanism (AttentionMechanism) model, a Transformer model, a GPT model, a visual geometry group (VGG) model, an OverFeat model, a densely connected network (DenseNet, such as DenseNet-1001), GoogLeNet and AlexNet, etc. However, the present disclosure is not limited to this, and the model can also be other types of artificial intelligence models.

[0071] Figure 6 A schematic diagram of a system for model training according to an example embodiment is shown. Figure 6 , a system for model training may include a host, a GPU, and a storage device. The host may include an unloading module and a pre-fetch manager, the storage device may include a dynamic random access memory DRAM and a flash memory NAND, and the training of the model is performed in the GPU. The storage device may be a memory-semantics SSD (MS SSD).

[0072] In the following example embodiments, the storage device is described using MS SSD as an example, but it should be understood that the present disclosure is not limited thereto. For example, the storage device may also be any storage device that provides hardware cache (eg, DRAM).

[0073] The present disclosure may adopt the following methods to avoid performance loss caused by data transfer between GPU and MS SSD to achieve better training throughput. In an embodiment, before model training, an offload module may be used to analyze the entire training process to derive an optimal offload plan. The analysis time is negligible compared to the training time of large models. In the same or another embodiment, when the offload work is about to be completed, the prefetch manager may schedule data prefetching from MS SSD NAND to MSSSD DRAM. Figure 6 The interaction between the host, GPU and MS SSD in one iteration of the model is shown. It should be understood that the training of the model may include multiple iterations, and each iteration may include forward propagation and back propagation. Figure 6As shown, the host can send an activation unload / load instruction to the GPU, and the GPU communicates with the MS SSD via the Compute Express Link (CXL) protocol. The activation is loaded from the DRAM of the MS SSD to the GPU memory through CXL.mem in the reverse propagation, and the activation is unloaded from the GPU memory to the NAND of the MS SSD through CXL.io in the forward propagation. After the unload / load is completed, the GPU returns the unload / load result to the host. In addition, in the reverse propagation, the host can send an activation prefetch notification to the MS SSD, and the MS SSD that receives the notification can perform activation prefetch from NAND to DRAM and return the prefetch result to the host.

[0074] Figure 7 A diagram showing the interaction between MS SSD internal prefetching and the host side according to an embodiment.

[0075] exist Figure 7 In the process, the host and MS SSD communicate via the CXL protocol. The CXL protocol is an interconnect protocol built on Peripheral Component Interconnect Express (PCIe) that integrates CPU, accelerator and memory devices into a computing domain. The CXL protocol also allows the host CPU to directly operate the device memory via load / store instructions. The high-performance mode of MS SSD includes the following functions: (1) data prefetch operation (hardware cache): prefetch is to preload data from NAND to DRAM based on host request; and (2) support for dual-mode access: NVMe read / write through CXL.io, and logical block address (LBA) memory read access through CXL.mem, with a read latency of microseconds (DRAM-like read latency: <1us under 100% cache hit). Table 2 shows the CXL.mem performance of MS SSD as follows:

[0076] Table 2

[0077]

[0078] like Figure 7 As shown, the host sends a prefetch request to the MS SSD via CXL.io, the MS SSD prefetches data from the NAND into the DRAM cache, and the data stored in the DRAM cache is loaded into the host memory via CXL.mem.

[0079] Figure 8 A flow chart illustrating a process for model training applied to a host according to an example embodiment.

[0080] The training of the model includes two processes: forward propagation and back propagation. In the forward propagation, the intermediate data (e.g., activation) generated by the calculation of the layer of the model is unloaded to the NAND of the storage device, and in the back propagation, the intermediate data of the layer to be calculated is pre-fetched from NAND to DRAM, and the intermediate data pre-fetched to DRAM is loaded into the memory of the GPU for calculation during the current layer calculation to solve the "memory wall" problem of the GPU and accelerate the training of large models.

[0081] First, we introduce data prefetching in back-propagation, in which the intermediate data has been unloaded from the GPU memory to the NAND of the storage device in the forward propagation.

[0082] In step S810, the graphics processor GPU is instructed to perform data loading, which loads the intermediate data of the current layer of the model being calculated from the dynamic random access memory DRAM of the storage device into the memory of the GPU.

[0083] In an example embodiment, in the reverse propagation, the intermediate data of the current layer of the model that is performing calculations has completed pre-fetching from the storage device NAND to the DRAM and is stored in the DRAM. For the calculation of the current layer, the host can instruct the GPU to perform data loading, and the GPU that receives the instruction can load the intermediate data of the current layer of the model from the DRAM of the storage device into the GPU memory, and the intermediate data of the current layer can be used by the GPU in the reverse propagation of the model to perform calculations for the current layer.

[0084] According to example embodiments, the intermediate data may include an activation value, and the calculation for the current layer may include a gradient calculation with respect to the activation value.

[0085] In step S820, based on the space capacity of the DRAM, the storage device is notified to perform a data pre-fetch operation. The data pre-fetch operation includes pre-fetching the intermediate data of the layer to be calculated of the model from the NAND of the storage device to the DRAM.

[0086] According to example embodiments, a data pre-fetch operation and a computing operation for a current layer may be performed at least partially in parallel.

[0087] In an example embodiment, while the GPU is performing a calculation operation for the current layer, the host may notify the storage device to perform a data prefetch operation based on the space capacity of the DRAM, that is, the data prefetch operation and the calculation operation for the current layer may be performed at least partially in parallel. The storage device that receives the notification performs a data prefetch operation to prefetch the intermediate data of the layer to be calculated of the model from the NAND of the storage device to the DRAM. In back propagation, the layer to be calculated is the layer before the current layer.

[0088] Fig. 9 A comparison diagram showing NVMe as backup storage and an MS SSD as backup storage according to an embodiment is shown.

[0089] Referring to Fig. 9 , in forward propagation (i.e., FP), the intermediate data (e.g., activation A [1] -A [n] ) generated by the calculation of the layers of the model is offloaded to the NAND of a traditional SSD or an MS SSD. And in backpropagation (i.e., BP), the current layer where the calculation is being performed is layer 4.

[0090] In the case of a traditional SSD as backup storage, in backpropagation, three steps are involved: (1) loading the intermediate data (e.g., activation A [i] ) of the current layer from the SSD NAND to the GPU memory; (2) the GPU uses the intermediate data of the current layer to perform the calculation for the current layer, e.g., calculating the gradient dA [i] of the current layer; (3) loading the intermediate data (e.g., activation A i-1 ) of the previous layer [i -1] . At this time, without prefetching, there is no GPU memory space, and only loading from the NAND is performed. The throughput of data loading is low and the latency is high.

[0091] In the case of an MS SSD as backup storage according to an exemplary embodiment, in backpropagation, the following four steps are involved: (1) loading the intermediate data (e.g., activation A [i] ) of the current layer from the MS SSD DRAM to the GPU memory; (2) the GPU uses the intermediate data of the current layer to perform the calculation for the current layer, e.g., calculating the gradient dA [i] of the current layer; (3) the host (e.g., the prefetch manager of the host) estimates the previous layer (i.e., the layer to be calculated) to be prefetched, which may include the host identifying the current layer and determining whether the previous layer to be prefetched meets the prefetch condition; (4) the host can asynchronously notify the MS SSD to prefetch the intermediate data (e.g., activation A [m] , m < i) of the previous layer from the NAND to the DRAM. The MS SSD that receives the notification performs the prefetch operation and the prefetch time overlaps with the calculation (e.g., gradient calculation) for the current layer. When the intermediate data of the current layer is loaded from the MS SSD to the GPU memory, its data prefetch has been completed in advance and stored in the DRAM. The loading of the intermediate data is equivalent to a 100% cache hit, with high throughput and low latency. In the case of an MS SSD as backup storage according to an exemplary embodiment, the loading speed of each layer is 3.5 times faster than that of a traditional SSD as backup storage.

[0092] According to an example embodiment, based on the space capacity of the DRAM, a storage device is notified to perform a data prefetch operation, including: traversing the layer to be calculated of the model as a prefetch layer, and based on the fact that there is remaining space in the DRAM and the data prefetch operation has not been performed for the prefetch layer, notifying the storage device to perform a data prefetch operation for the prefetch layer.

[0093] Fig.10 A flow chart of a data pre-fetch process from MS SSD NAND to DRAM in reverse propagation according to an example embodiment is shown.

[0094] In operation S1010, it is determined whether the current layer i is greater than 0. If it is "no", the last layer of the model has been traversed and the process ends. If it is "yes", it proceeds to operation S1020.

[0095] In operation S1020, the current layer intermediate data is loaded (eg, activation A [i] ) to the GPU memory. The host instructs the GPU to perform data loading, and the GPU that receives the instruction loads the intermediate data from the DRAM of the storage device (e.g., MS SSD) to the GPU memory. For example, the GPU can read the intermediate data stored in the DRAM of the storage device to the GPU memory based on CXL.mem. Specifically, the GPU can send a read request to the storage device, and then the storage device sends the intermediate data of the current layer stored in the DRAM in response to the read request to be loaded into the GPU memory.

[0096] In operation S1030, the current layer gradient is calculated. The GPU uses the intermediate data of the current layer to perform calculations on the current layer, such as calculating dA [i] .

[0097] In operation S1040, loop parameters m=i-1, sum_size=0 are initialized, where m represents a prefetch layer, and sum_size represents the total size of prefetched intermediate data (eg, activations). The host traverses the layers to be calculated of the model as prefetch layers and performs data prefetch operations for the prefetch layers.

[0098] In operation S1050, it is determined whether the pre-fetch layer satisfies the pre-fetch condition m>0, A [m] Not in DRAM (ie, data prefetching has not been performed for this prefetch layer) and sum_size <size DRAM (i.e., DRAM has spare space), where size DRAM Indicates the size of the DRAM of the storage device (eg, MS SSD). The host determines whether the prefetch layer satisfies the prefetch condition. If "no", no data prefetch operation is performed for the prefetch layer, and the process proceeds to operation S1070. If "yes", the process proceeds to operation S1060.

[0099] In operation S1060, the storage device is asynchronously notified to pre-fetch the previous layer (ie, the m layer, the pre-fetch layer) intermediate data from the NAND (eg, activate A [m] ) to DRAM, sum_size=sum_size+A size [m] And m=m-1. The host notifies the MS SSD to perform the data pre-fetch operation and returns to operation S1050, and the MS SSD that receives the notification performs the data pre-fetch operation. The loop execution of operations S1050 and S1060 can pre-fetch intermediate data of multiple previous layers (i.e., layers to be calculated) during one calculation for the current layer of the model.

[0100] In operation S1070, the current layer i=i-1 is updated, and the process returns to operation S1010.

[0101] The operations of the above process involve operations of a host, a GPU, and a storage device, and the operations are not limited to being executed sequentially, but may also be executed concurrently. For example, operation S1040 may be executed concurrently with operations S1050 and S1060.

[0102] Fig.11 A diagram illustrating a process of pre-fetching offloaded intermediate data in backward propagation according to example embodiments.

[0103] In backpropagation, the GPU uses the intermediate data of the current layer (e.g., activation) to perform calculations for the current layer (e.g., calculate gradients). Fig.11 , the current layer in the GPU that is performing calculations is layer 3, layers 1-2 are previous layers or layers to be calculated, and layers 4-7 are layers that have completed calculations. During backpropagation, a gradient calculation flow for the GPU and a previous layer prefetch flow for the MSSSD are included. The prefetch algorithm overlaps the current layer calculation with the prefetch of the previous layer, and prefetches as many activations as possible from NAND to DRAM, prefetching A [i] Count ≈ size DRAM / A [i] avg_size In an example embodiment, an A may be pre-fetched during a computation for a layer of the model. [i] For example, in the calculation for layer 6, layer 5, layer 4 and layer 3, the corresponding pre-fetch A [5] , A [4] , A [3] and A [2] ; It is also possible to pre-fetch multiple A during one computation for a layer of the model [i] For example, in the calculation for layer 6, layer 5, layer 4 and layer 3, the corresponding pre-fetch A [5] and A [4] , A[3] , A [2] and A [1] , where A is pre-fetched from NAND in the calculation for layer 6 [5] and A [4] Two are activated into DRAM.

[0104] Next, we will introduce data unloading in forward propagation. The intermediate data loaded in reverse propagation are all the intermediate data unloaded in forward propagation.

[0105] According to an example embodiment, in the forward propagation of the model, the GPU may be instructed to perform data offloading, which offloads intermediate data generated by calculations for the layers of the model from the memory of the GPU to the NAND.

[0106] Fig.12 A diagram illustrating an offloading process using a MS SSD as a backup storage according to an example embodiment.

[0107] Reference Fig.12 In the forward propagation (FP), the offloading process may include four steps: (1) the host uses the offloading module to derive the optimal offloading solution (e.g., offloading A [1] , A [2] , A [3] , A [4] ...A [n-1] , A [n] ); (2) The GPU computes the current layer and generates intermediate data (e.g., activation A [i] ); (3) After the calculation of the current layer is completed, the intermediate data is unloaded from the GPU memory to the NAND of the storage device (e.g., MS SSD), wherein the host can instruct the GPU to perform data unloading. The GPU receiving the host instruction unloads the intermediate data from the GPU memory to the NAND of the storage device. Accordingly, the storage device receives the intermediate data unloaded from the GPU memory to store in the NAND, where the data unloading between the GPU and the storage device can be based on CXL.io; and (4) come to step (2) until the nth layer of the model.

[0108] According to an example embodiment, before model training, information of a layer in a plurality of layers of a model that unloads intermediate data to NAND may be obtained based on the amount of memory of the GPU and the execution time of the forward propagation. In the forward propagation of the model, instructing the GPU to perform data unloading may include: instructing the GPU to perform data unloading in the forward propagation of the model based on the information of the layer.

[0109] According to an example embodiment, information of the layer may be derived based on the fact that the total amount of remaining intermediate data is less than or equal to the amount of memory of the GPU and the total unloading time is less than or equal to the execution time of the forward propagation. The total amount of remaining intermediate data is based on the amount of intermediate data of the layers other than the layer that unloads the intermediate data to the NAND among the multiple layers of the model. The total unloading time is based on the unloading time of the layer that unloads the intermediate data to the NAND.

[0110] In an example embodiment, the intermediate data may be unloaded on the basis that the sum of the intermediate data (e.g., activation) generated by the calculation layers of the model and the total size of other memories is greater than or equal to the amount of GPU memory. That is, in order to ensure trainability, the following formula 1 needs to be satisfied:

[0111] A1 size + A2 size + … + An size + Non-offload size >= GPU size Formula 1

[0112] Among them, An size Means: the size of the nth offloaded activation tensor, GPU size : Indicates the amount of GPU memory, Non-offload size : The total size of other memory (i.e., resident objects such as input, temporary workspace, etc.).

[0113] In addition, on the basis of satisfying the following formula 2, information of a layer in the multiple layers of the model that unloads the intermediate data to the NAND, that is, an unloading scheme or unloading schedule is derived:

[0114] A1 time + A2 time + … + An time <= FP time Formula 2

[0115] Among them, An time Meaning: Unloading time of the nth activation tensor, FP time represents: the execution time of the forward pass (excluding offloading time).

[0116] Fig.13 FIG. 2 shows a process for deriving an optimal unloading schedule according to an example embodiment. Fig.13 , an initial state and three uninstallation schemes are shown, namely uninstallation schemes 1-3.

[0117] Initial state: GPU memory size: GPU size =8MB, A1~A7 represents layer (1) ~ Layer (7)The activation tensor of A1 size ~A7 size They are 6MB, 2MB, 4MB, 4MB, 4MB, 2MB and 2MB respectively. Exceeding GPU memory (overflow amount) = A1 size +A2 size +A3 size +A4 size +A5 size +A6 size +A7 size -GPU size =16MB.

[0118] Uninstallation plan 1: Uninstall size: A1 size +A2 size =8MB, GPU memory exceeded after unloading: 8MB, unloading time: A1 time +A2 time <FP time ; or uninstall size: A1 size +A2 size +A3 size +A4 size =16MB, GPU memory exceeded after uninstallation: 0MB, uninstallation time: A1 time +A2 time +A3 time +A4 time >FP time .

[0119] Uninstallation plan 2: Uninstall size: A1 size +A2 size +A3 size +A5 size =16MB, GPU memory exceeded after uninstallation: 0MB, uninstallation time: A1 time +A2 time +A3 time +A5 time >FP time .

[0120] Uninstallation plan 3: Uninstall size: A1 size +A3 size +A5 size +A6 size =16MB, GPU memory exceeded after uninstallation: 0MB, uninstallation time: A1 time +A3 time +A5 time +A6 time =FP time .

[0121] Fig.13The offloading schemes 1 and 2 in [ ] may not be considered as the optimal offloading schemes, while the offloading scheme 3 may be the optimal offloading scheme. In the offloading scheme 3, the total amount of the remaining activation tensors (i.e., the total amount of the remaining intermediate data) does not exceed the GPU memory capacity and the offloading time does not exceed the execution time of the forward propagation. In the offloading scheme 3, the total amount of the remaining activation tensors includes the sum of the sizes of the activation tensors of the layers in the multiple layers of the model except for the layer that offloads the activation tensors to the NAND. For example, in the offloading scheme 3, the total amount of the remaining activation tensors = A2 size + A4 size + A7 size = 8MB = GPU size . Fig.13 The selective offloading process in [ ] is repeated until an optimal offloading scheme is obtained. If there is no optimal offloading scheme, a sub-optimal offloading scheme is selected to ensure trainability. For example, the optimal offloading scheme may be a scheme in which the total amount of the remaining activation tensors is equal to the GPU memory capacity and the offloading time is equal to the execution time of the forward propagation.

[0122] In an embodiment of the present disclosure, before the model training, the host derives the information of the layer that offloads the intermediate data to the NAND in the multiple layers of the model. When the model is being trained, the information of the layer (e.g., offloading indication) can be indicated to the GPU (once per iteration) in the forward propagation of each iteration of the model, and the GPU performs the offloading of the corresponding layer of the iteration based on the information of the layer. The CPU can also indicate the GPU to perform offloading (multiple indications per iteration) after the calculation of the corresponding layer according to the information of the layer in the forward propagation of each iteration of the model, and the GPU performs the offloading of the corresponding layer based on the indication. In addition, for the layer that performs data offloading in the forward propagation, data prefetching and data loading operations are performed for this layer in the backward propagation.

[0123] Fig.14 FIG. shows a flowchart of a process for model training applied to a storage device according to an exemplary embodiment. The storage device may include DRAM and NAND.

[0124] In step S1410, a notification for performing a data prefetch operation is received from the host.

[0125] In an exemplary embodiment, in the backward propagation, the host may asynchronously notify the storage device (e.g., MS SSD) to prefetch the intermediate data (e.g., activation A [m] , where i is the current layer and m < i) of the previous layer (i.e., the layer to be calculated) from the NAND to the DRAM.

[0126] In step S1420, based on the received notification, a data prefetch operation is performed, where the data prefetch operation includes prefetching the intermediate data of the layer to be calculated of the model from the NAND to the DRAM.

[0127] In an example embodiment, the storage device (eg, MS SSD) receiving the notification executes the process of converting the intermediate data of the previous layer (eg, activating A [m] )Prefetch operation from NAND to DRAM.

[0128] According to an example embodiment, the method may further include: sending intermediate data of a current layer of the model that is performing calculations stored in DRAM to be loaded into a memory of a graphics processor GPU, wherein the intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer. Specifically, based on the computational fast link CXL protocol, the intermediate data of the current layer stored in DRAM is sent to be loaded into the memory of the graphics processor GPU.

[0129] According to example embodiments, the data pre-fetch operation may be performed at least partially in parallel with the computation operation for the current layer.

[0130] In the example embodiment, during the back propagation, the host instructs the GPU to perform data loading, and the GPU receiving the instruction loads the intermediate data of the current layer (e.g., activation A [i] , i is the current layer) is loaded from the DRAM of the storage device (e.g., MS SSD) to the GPU memory. For example, the GPU can read the intermediate data of the current layer stored in the DRAM of the storage device to the GPU memory based on CXL.mem. Specifically, the GPU can send a read request to the storage device, and then the storage device sends the intermediate data of the current layer stored in the DRAM in response to the read request to be loaded into the GPU memory. Then, the GPU uses the intermediate data of the current layer in the back propagation of the model to perform calculations for the current layer (e.g., including gradient calculations on activations, dA [i] ).

[0131] In an example embodiment, in the back propagation, a gradient calculation flow for a GPU and a previous layer prefetch flow for a storage device may be included, and the current layer calculation may be overlapped with the prefetch of the previous layer.

[0132] According to an example embodiment, the method may further include: in the forward propagation of the model, receiving intermediate data generated by the calculation of the layer of the model unloaded from the memory of the GPU to be stored in the NAND. Specifically, based on the computation fast link CXL protocol, receiving the intermediate data generated by the calculation unloaded from the memory of the GPU to be stored in the NAND.

[0133] In an example embodiment, in forward propagation, the host may instruct the GPU to perform data offloading, and the GPU receiving the host's instruction unloads the intermediate data from the GPU memory to the NAND of the storage device. Accordingly, the storage device receives the intermediate data unloaded from the GPU memory to store in the NAND. Here, the data offloading between the GPU and the storage device may be based on CXL.io.

[0134] The method for model training applied to the host and storage device as described above utilizes data offloading between the GPU and the storage device to solve the "memory wall" problem of the GPU, and utilizes data prefetching to speed up model training, thereby achieving better training throughput; because the GPU and the storage device perform data transmission through direct communication, less CPU resources and main memory bandwidth can be used and robust performance can be provided; in addition, while the software implementation complexity is low, higher storage capacity and lower cost are provided for large model training.

[0135] Fig.15 A schematic diagram illustrating a host according to an example embodiment.

[0136] Reference Fig.15 , the host 1500 may include a memory 1510 and a processor 1520, wherein the memory 1520 may store instructions. When the instructions are executed by the processor 1520, the processor 1520 may be prompted to perform the following operations: instruct the graphics processor GPU to perform data loading, and the data loading loads the intermediate data of the current layer of the model being calculated from the dynamic random access memory DRAM of the storage device to the memory of the GPU. The instructions may prompt the processor to notify the storage device to perform a data prefetch operation based on the space capacity of the DRAM, and the data prefetch operation includes prefetching the intermediate data of the layer to be calculated of the model from the NAND of the storage device to the DRAM. The intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer.

[0137] According to example embodiments, the data pre-fetch operation may be performed at least partially in parallel with the computation operation for the current layer.

[0138] According to an example embodiment, when the instruction is executed by processor 1520, it may cause processor 1520 to perform the following operations: traverse the layers to be calculated of the model as prefetch layers, and based on the fact that there is remaining space in DRAM and data prefetch operations have not been performed for the prefetch layers, notify the storage device to perform data prefetch operations for the prefetch layers.

[0139] According to an example embodiment, when the instruction is executed by the processor 1520, the processor 1520 may be prompted to perform the following operations: in the forward propagation of the model, instruct the GPU to perform data unloading, which unloads the intermediate data generated by the calculation of the layers of the model from the GPU's memory to the NAND.

[0140] According to an example embodiment, when the instruction is executed by the processor 1520, the processor 1520 may be prompted to perform the following operations: before model training, based on the memory amount of the GPU and the execution time of the forward propagation, information of the layer in the multiple layers of the model that unloads the intermediate data to the NAND is obtained. When the instruction is executed by the processor 1520, the processor may be prompted to control or instruct the GPU to perform data unloading in the forward propagation of the model based on the information of the layer.

[0141] According to an example embodiment, when the instruction is executed by the processor 1520, the processor 1520 may be caused to perform the following operations: based on the total amount of remaining intermediate data being less than or equal to the amount of memory of the GPU and the total unloading time being less than or equal to the execution time of the forward propagation, deriving information of the layer. The total amount of remaining intermediate data is based on the amount of intermediate data of the layers other than the layer that unloads the intermediate data to the NAND among the multiple layers of the model. The total unloading time is based on the unloading time of the layer that unloads the intermediate data to the NAND.

[0142] According to example embodiments, the intermediate data may include an activation value, and the calculation for the current layer may include a gradient calculation with respect to the activation value.

[0143] Fig.16 A schematic diagram illustrating a storage device according to an example embodiment.

[0144] Reference Fig.16 The storage device 1600 may include a dynamic random access memory DRAM 1610 and a NAND 1620. The storage device 1600 may receive a notification of performing a data prefetch operation from a host, and based on the received notification, perform a data prefetch operation, wherein the data prefetch operation includes prefetching intermediate data of a layer to be calculated of the model from the NAND 1620 to the DRAM 1610.

[0145] According to an example embodiment, the storage device 1600 may send intermediate data of a current layer of a model that is performing calculations stored in the DRAM 1610 to be loaded into the memory of a graphics processor GPU, wherein the intermediate data of the current layer is used by the GPU in back propagation of the model to perform calculations for the current layer.

[0146] According to example embodiments, the data pre-fetch operation may be performed at least partially in parallel with the computation operation for the current layer.

[0147] According to an example embodiment, the storage device 1600 may receive intermediate data generated by calculation for a layer of a model unloaded from a memory of a GPU to be stored in the NAND 1620 in the forward propagation of the model.

[0148] According to an example embodiment, the storage device 1600 may send intermediate data of the current layer stored in the DRAM 1610 to be loaded into the memory of the graphics processor GPU based on the CXL protocol, and receive intermediate data generated by calculations unloaded from the memory of the GPU to be stored in the NAND 1620 based on the CXL protocol.

[0149] According to example embodiments, the storage device 1600 may include a memory-semantic solid state drive MS SSD.

[0150] The host and storage device described above utilize data offloading between the GPU and the storage device to solve the GPU's "memory wall" problem, and utilize data prefetching to speed up model training, thereby achieving better training throughput. Because the GPU and storage device perform data transmission through direct communication, less CPU resources and main memory bandwidth can be used and robust performance can be provided. In addition, while the software implementation complexity is low, higher storage capacity and lower cost are provided for large model training.

[0151] The advantages and disadvantages of the present disclosure and the existing solutions are shown in Table 3 below:

[0152] Table 3

[0153]

[0154]

[0155] The method, host and storage device for model training disclosed in the present invention can provide higher training throughput based on the high performance of MS SSD by effectively unloading and loading intermediate data during the training process compared with traditional SSD. Any large model (e.g., DNN) training can use it through a simple software stack implementation. Since there are fewer CPU resources and memory bandwidth competing with the CPU process, robust performance can be provided. In addition, it provides larger storage capacity and lower cost than GPU HBM (High Bandwidth Memory) DRAM and CPU DRAM (where HBM DRAM: $20 / GB, DDR4 DRAM: $3.6 / GB and flash NAND: $0.102 / GB), and reduces training TCO. In short, the present disclosure provides larger storage capacity and lower cost for training large models. The present disclosure can be used for large model training based on memory capacity and bandwidth.

[0156] For example, when the training batch size is only set to 1, training a trillion-parameter model will generate more than 1TB of activation tensors (i.e., intermediate data). By buffering (i.e., unloading) 1TB of activation tensors, the existing solution is compared with the present disclosure, and the specific effects are shown in Table 4-5:

[0157] Table 4

[0158] Data transmission delay(s) Offload activation to NVMe SSD (7GB / s) 146 This disclosure (25.5GB / s) 40

[0159] Table 5

[0160] Storage cost ($) Using distributed GPU servers 20480 Offload activation to CPU memory 3686.4 The Disclosure 104.448

[0161] Fig.17 Schematic diagram of a system 1000 to which a storage device is applied according to an embodiment.

[0162] Fig.17 The system 1000 may be, for example, a mobile system such as a portable communication terminal (e.g., a mobile phone), a smart phone, a tablet personal computer (PC), a wearable device, a healthcare device, or an Internet of Things (IOT) device. However, Fig.17 The system 1000 is not limited thereto, and may be, for example, a PC, a laptop computer, a server, a media player, or an automotive device (eg, a navigation device).

[0163] Reference Fig.17 , the system 1000 may include a main processor 1100, a memory (e.g., 1200a and 1200b), and a storage device (e.g., 1300a and 1300b). In addition, the system 1000 may include at least one of an image capture device 1410, a user input device 1420, a sensor 1430, a communication device 1440, a display 1450, a speaker 1460, a power supply device 1470, and a connection interface 1480.

[0164] The main processor 1100 may control all operations of the system 1000, including, for example, operations of other components included in the system 1000. The main processor 1100 may be implemented as, for example, a general-purpose processor, a dedicated processor, an application processor, or the like.

[0165] The main processor 1100 may include at least one central processing unit (CPU) core 1110 and a controller 1120 for controlling memories 1200a and 1200b and / or storage devices 1300a and 1300b. In some exemplary embodiments, the main processor 1100 may further include an accelerator 1130, which is a dedicated circuit for high-speed data operations such as, for example, artificial intelligence (AI) data operations. The accelerator 1130 may include, for example, a graphics processing unit (GPU), a neural processing unit (NPU), and / or a data processing unit (DPU), etc., and may be implemented as a chip physically separated from other components of the main processor 1100.

[0166] The memories 1200a and 1200b may be used as main storage devices of the system 1000. Although the memories 1200a and 1200b may respectively include volatile memories such as, for example, static random access memory (SRAM) and / or dynamic random access memory (DRAM), etc., according to an embodiment, the memories 1200a and 1200b may respectively include non-volatile memories such as, for example, flash memory, phase change random access memory (PRAM) and / or resistive random access memory (RRAM), etc. The memories 1200a and 1200b may be implemented in the same package as the main processor 1100.

[0167] The storage devices 1300a and 1300b may be used as non-volatile storage devices, which are configured to store data regardless of whether they are powered, and may have a larger storage capacity than the memories 1200a and 1200b. The storage devices 1300a and 1300b may include memory controllers (STRG CTRL) 1310a and 1310b and non-volatile memories (NVM) 1320a and 1320b, respectively, which are configured to store data under the control of the memory controllers 1310a and 1310b. Although the NVMs 1320a and 1320b may include V-NAND flash memories having a two-dimensional (2D) or three-dimensional (3D) structure, the NVMs 1320a and 1320b may include other types of NVMs, such as, for example, PRAM and / or RRAM, etc.

[0168] The storage devices 1300a and 1300b may be physically separated from the main processor 1100 and included in the system 1000, or may be implemented in the same package as the main processor 1100. The storage devices 1300a and 1300b may be solid-state devices (SSDs) or memory cards, and may be removably combined with other components of the system 100 through interfaces such as the connection interface 1480 described further later. The storage devices 1300a and 1300b may be devices to which standard protocols such as, for example, universal flash memory (UFS), embedded multimedia card (eMMC), or NVMe are applied. However, the storage devices 1300a and 1300b are not limited thereto.

[0169] The image capture device 1410 can capture still images or moving images. The image capture device 1410 may include, for example, a camera, a camcorder, and / or a webcam.

[0170] The user input device 1420 may receive various types of data input by a user of the system 1000 and may include, for example, a touch pad, a keypad, a keyboard, a mouse, a microphone, and the like.

[0171] The sensor 1430 may detect various types of physical quantities that may be obtained from the outside of the system 1000 and convert the detected physical quantities into electrical signals. The sensor 1430 may include, for example, a temperature sensor, a pressure sensor, an illumination sensor, a position sensor, an acceleration sensor, a biosensor, and / or a gyro sensor.

[0172] The communication device 1440 may transmit and receive signals between other devices outside the system 1000 according to various communication protocols. The communication device 1440 may include, for example, an antenna, a transceiver, or a modem.

[0173] Display 1450 and speaker 1460 may be used as output devices configured to output visual information and auditory information, respectively, to a user of system 1000 .

[0174] The power supply device 1470 may appropriately convert power supplied from a battery embedded in the system 1000 and / or an external power source, and supply the converted power to each component of the system 1000 .

[0175] The connection interface 1480 may provide a connection between the system 1000 and an external device that is connected to the system 1000 and is capable of transmitting and receiving data to and from the system 1000. The connection interface 1480 may be implemented by using various interface schemes, such as, for example, Advanced Technology Attachment (ATA), Serial ATA (SATA), External Serial ATA (e-SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Peripheral Component Interconnect (PCI), PCI express (PCIe), NVMe, IEEE 1394, Universal Serial Bus (USB) interface, Secure Digital (SD) card interface, Multimedia Card (MMC) interface, Embedded Multimedia Card (eMMC) interface, UFS interface, Embedded UFS (eUFS) interface, Compact Flash (CF) card interface, and the like.

[0176] According to an embodiment of the present disclosure, a system (e.g., 1000) applying a storage device is provided, including: a main processor (e.g., 1100); a memory (e.g., 1200a and 1200b); and a storage device (e.g., 1300a and 1300b), wherein the storage device is configured to execute the method for model training as described above.

[0177] Fig.18 is a block diagram of a host storage system 10 according to an embodiment.

[0178] The host storage system 10 may include a host 100 and a storage device 200. The storage device 200 may include a memory controller 210 (in Fig.18 According to an embodiment, the host 100 may include a host controller 110 and a host memory 120. The host memory 120 may be used as a buffer memory configured to temporarily store data to be transmitted to or received from the storage device 200.

[0179] The storage device 200 may include a storage medium configured to store data in response to a request from the host 100. As an example, the storage device 200 may include at least one of an SSD, an embedded memory, and a removable external memory. When the storage device 200 is an SSD, the storage device 200 may be a device that complies with the NVMe standard. When the storage device 200 is an embedded memory or an external memory, the storage device 200 may be a device that complies with the UFS standard or the eMMC standard. Both the host 100 and the storage device 200 may generate a packet according to the adopted standard protocol and may send the packet.

[0180] When the NVM 220 of the storage device 200 includes a flash memory, the flash memory may include a 2D NAND memory array or a 3D (or vertical) NAND (VNAND) memory array. As another example, the storage device 200 may include various other types of NVM. For example, the storage device 200 may include a magnetic random access memory (MRAM), a spin transfer torque MRAM, a conductive bridge RAM (CBRAM), a ferroelectric RAM (FRAM), a PRAM, a RRAM, and various other types of memory.

[0181] According to an exemplary embodiment, the host controller 110 and the host memory 120 may be implemented as separate semiconductor chips. Alternatively, in some exemplary embodiments, the host controller 110 and the host memory 120 may be integrated in the same semiconductor chip. As an example, the host controller 110 may be any one of a plurality of devices included in an application processor (AP). The AP may be implemented as, for example, a system on chip (SoC). In addition, the host memory 120 may be an embedded memory included in the AP or a memory device external to the AP.

[0182] The host controller 110 may manage an operation of storing data (eg, write data) of the buffer area of ​​the host memory 120 in the NVM 220 or an operation of storing data (eg, read data) of the NVM 220 in the buffer area.

[0183] The memory controller 210 may include a host interface 211, a memory interface 212, and a CPU 213, a flash translation layer (FTL), a package manager 215 (in Fig.18 The memory controller 210 may further include a working memory in which the FTL 214 is loaded. The CPU 213 may execute the FTL 214 to control data write and read operations on the NVM 220.

[0184] The host interface 211 may transmit and receive packets to and from the host 100. The packets transmitted from the host 100 to the host interface 211 may include commands or data to be written to the NVM 220, etc. The packets transmitted from the host interface 211 to the host 100 may include responses to commands or data read from the NVM 220, etc. The memory interface 212 may transmit data to be written to the NVM 220 to the NVM 220 or receive data read from the NVM 220. The memory interface 212 may be configured to comply with a standard protocol such as, for example, Toggle or Open NAND Flash Interface (ONFI).

[0185] The FTL 214 may perform various functions, such as, for example, an address mapping operation, a wear leveling operation, and a garbage collection operation. The address mapping operation may be an operation of converting a logical address received from the host 100 into a physical address for actually storing data in the NVM 220. The wear leveling operation may be a technique for preventing or reducing excessive degradation of a specific block by allowing the blocks of the NVM 220 to be used evenly. As an example, the wear leveling operation may be implemented by using a firmware technique that balances the erase counts of physical blocks. The garbage collection operation may be a technique for ensuring available capacity in the NVM 220 by erasing an existing block after copying valid data of the existing block to a new block.

[0186] The packet manager 215 may generate a packet according to a protocol agreed with an interface of the host 100, or parse various types of information from a packet received from the host 100. In addition, the buffer memory 216 may temporarily store data to be written to or read from the NVM 220. Although the buffer memory 216 may be a component included in the memory controller 210, in an embodiment, the buffer memory 216 may be provided outside the memory controller 210.

[0187] The ECC engine 217 may perform error detection and correction operations on the read data read from the NVM 220. For example, the ECC engine 217 may generate parity bits for write data to be written to the NVM 220, and the generated parity bits may be stored together with the write data in the NVM 220. During the reading of data from the NVM 220, the ECC engine 217 may correct errors in the read data by using the read data and the parity bits read from the NVM 220, and output the error-corrected read data.

[0188] The AES engine 218 may perform at least one of an encryption operation and a decryption operation on data input to the memory controller 210 by using a symmetric key algorithm.

[0189] According to an embodiment of the present disclosure, a host storage system (e.g., 10) is provided, comprising: a host (e.g., 100); and a storage device (200), wherein the storage device is configured to execute the method for model training as described above.

[0190] Fig.19 is a diagram of a data center 3000 to which a storage device is applied according to an exemplary embodiment.

[0191] Application / Storage Servers

[0192] Reference Fig.19, the data center 3000 may be a facility that collects various types of data and provides services, and may be referred to as a data storage center. The data center 3000 may be a system for operating a search engine and a database, and may be a computing system used by a company (such as a bank) or a government agency. The data center 3000 may include application servers 3100 to 3100n and storage servers 3200 to 3200m, where n and m are positive integers. According to an exemplary embodiment, the number of applications 3100 to 3100n and the number of storage servers 3200 to 3200m may be selected differently. The number of application servers 3100 to 3100n and the number of storage servers 3200 to 3200m may be different from each other.

[0193] The application server 3100 or the storage server 3200 may include processors 3110 and 3210 and at least one of memories 3120 and 3220, at least one of switches 3130 to 3130n, at least one of network interface cards (NICs) 3140 to 3140n and 3240 to 3240m, at least one of DRAMs 3253 to 3253m, and at least one of controllers 3251 to 3251m. The storage server 3200 will now be described as an example. The processor 3210 may control all operations of the storage server 3200, access the memory 3220, and execute instructions and / or data loaded into the memory 3220. The memory 3220 may be, for example, a double data rate synchronous DRAM (DDR SDRAM), a high bandwidth memory (HBM), a hybrid memory cube (HMC), a dual in-line memory module (DIMM), an Optane DIMM (Optane DIMM), or a memory 3220. TM DIMM) or non-volatile DIMM (NVMDIMM). In some exemplary embodiments, the number of processors 3210 and memories 3220 included in the storage server 3200 may be selected differently. In an exemplary embodiment, the processor 3210 and the memory 3220 may provide a processor-memory pair. In an exemplary embodiment, the number of processors 3210 and the number of memories 3220 may be different from each other. The processor 3210 may include a single-core processor or a multi-core processor. The above description of the storage server 3200 may be similarly applied to the application server 3100. In some embodiments, the application server 3100 may not include a storage device 3150. The storage server 3200 may include at least one storage device 3250. According to an embodiment, the number of storage devices 3250 included in the storage server 3200 may be selected differently.

[0194] network

[0195] The application servers 3100 to 3100n can communicate with the storage servers 3200 to 3200m through the network 3300. The network 3300 can be implemented by using a fiber channel (FC) or Ethernet. In this case, FC can be a medium for relatively high-speed data transmission, and an optical switch with high performance and high availability can be used. According to the access method of the network 3300, the storage servers 3200 to 3200m can be set to file storage, block storage, or object storage.

[0196] In one embodiment, the network 3300 may be a network dedicated to storage, such as a storage area network (SAN). For example, the SAN may be an FC-SAN, which uses an FC network and is implemented according to the FC protocol (FCP). As another example, the SAN may be an Internet Protocol (IP)-SAN, which uses a transmission control protocol (TCP) / IP network and is implemented according to a SCSI or Internet SCSI (iSCSI) protocol over TCP / IP. In one embodiment, the network 3300 may be a general network, such as a TCP / IP network. For example, the network 3300 may be implemented according to protocols such as FC over Ethernet (FCoE), network attached storage (NAS), and NVMe over fabric (NVMe-oF).

[0197] Hereinafter, the application server 3100 and the storage server 3200 will be mainly described. The description of the application server 3100 may be applied to another application server 3100n, and the description of the storage server 3200 may be applied to another storage server 3200m.

[0198] The application server 3100 may store data requested to be stored by a user or client in one of the storage servers 3200 to 3200m through the network 3300. In addition, the application server 3100 may obtain data requested to be read by a user or client from one of the storage servers 3200 to 3200m through the network 3300. For example, the application server 3100 may be implemented as a network server or a database management system (DBMS).

[0199] The application server 3100 may access a memory 3120n or a storage device 3150n included in another application server 3100n through the network 3300. Alternatively, the application server 3100 may access a memory 3220 to 3220m or a storage device 3250 to 3250m included in the storage servers 3200 to 3200m through the network 3300. Therefore, the application server 3100 may perform various operations on the data stored in the application servers 3100 to 3100n and / or the storage servers 3200 to 3200m. For example, the application server 3100 may execute an instruction for moving or copying data between the application servers 3100 to 3100n and / or the storage servers 3200 to 3200m. In this case, data may be moved from the storage devices 3250 to 3250m of the storage servers 3200 to 3200m through the storage 3220 to 3220m of the storage servers 3200 to 3200m or directly to the storage 3120 to 3120n of the application servers 3100 to 3100n. The data moved through the network 3300 may be data encrypted for security or privacy.

[0200] Interface structure / type

[0201] The storage server 3200 will now be described as an example. The interface 3254 can provide a physical connection between the processor 3210 and the controller 3251 and a physical connection between the network interface card (NIC) 3240 and the controller 3251. For example, the interface 3254 can be implemented using a direct attached storage (DAS) solution, where the storage device 3250 is directly connected to a dedicated cable. For example, the interface 3254 can be implemented using various interface solutions, such as ATA, SATA, e-SATA, SCSI, SAS, PCI, PCIe, NVMe, IEEE 1394, USB interface, SD card interface, MMC interface, eMMC interface, UFS interface, eUFS interface, and CF card interface.

[0202] The storage server 3200 may further include a switch 3230 and a network interconnect (NIC) 3240. The switch 3230 may selectively connect the processor 3210 to the storage device 3250 under the control of the processor 3210, or selectively connect the NIC 3240 to the storage device 3250.

[0203] In one embodiment, the NIC 3240 may include a network interface card and a network adapter. The NIC 3240 may be connected to the network 3300 through, for example, a wired interface, a wireless interface, a Bluetooth interface, or an optical interface. The NIC 3240 may include an internal memory, a digital signal processor (DSP), and a host bus interface, and is connected to the processor 3210 and / or the switch 3230 through the host bus interface. The host bus interface may be implemented as one of the above examples of the interface 3254. In one embodiment, the NIC 3240 may be integrated with at least one of the processor 3210, the switch 3230, and the storage device 3250.

[0204] Interface Operation

[0205] In the storage servers 3200 to 3200m or the application servers 3100 to 3100n, the processor may send a command to the storage devices 3150 to 3150n and 3250 to 3250m or the memories 3120 to 3120n and 3220 to 3220m and program or read data. In this case, the data may be data in which an error is corrected by an ECC engine. The data may be data on which a data bus inversion (DBI) operation or a data masking (DM) operation is performed, and may include cyclic redundancy coding (CRC) information. The data may be data encrypted for security or privacy.

[0206] The storage devices 3150 to 3150n and 3250 to 3250m may send control signals and command / address signals to the NAND flash memory devices 3252 to 3252m in response to a read command received from the processor. Therefore, when data is read from the NAND flash memory devices 3252 to 3252m, a read enable (RE) signal may be input as a data output control signal, and thus, the data may be output to the DQ bus. The data strobe signal DQS may be generated using the RE signal. Depending on the rising edge or falling edge of the write enable (WE) signal, the command and address signals may be latched in the page buffer.

[0207] SSD Operations

[0208] The controller 3251 may control all operations of the storage device 3250. In one embodiment, the controller 3251 may include an SRAM. The controller 3251 may write data to the NAND flash memory device 3252 in response to a write command, or read data from the NAND flash memory device 3252 in response to a read command. For example, a write command and / or a read command may be provided from a processor 3210 of the storage server 3200, a processor 3210m of another storage server 3200m, or processors 3110 and 3110n of application servers 3100 and 3100n. The DRAM 3253 may temporarily store (or buffer) data to be written to the NAND flash memory device 3252 or data to be read from the NAND flash memory device 3252. And, the DRAM 3253 may store metadata. Here, the metadata may be user data or data generated by the controller 3251 for managing the NAND flash memory device 3252. The storage device 3250 may include a secure element (SE) for security or privacy.

[0209] According to an exemplary embodiment of the present disclosure, a data center system (e.g., 3000) is provided, comprising: a plurality of application servers (3100 to 3100n); and a plurality of storage servers (e.g., 3200 to 3200m), wherein each storage server comprises a storage device, wherein the storage device is configured to execute the method for model training as described above.

[0210] As is conventional in the field of the present disclosure, embodiments are described and shown in the accompanying drawings according to functional blocks, units and / or modules. It will be appreciated by those skilled in the art that these blocks, units and / or modules are physically implemented by electronic (or optical) circuits (such as logic circuits, discrete components, microprocessors, hard-wired circuits, memory elements, wiring connections, etc.), which can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. In the case where blocks, units and / or modules are implemented by microprocessors or the like, they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and they can be optionally driven by firmware and / or software. Optionally, each block, unit and / or module can be implemented by dedicated hardware, or implemented as a combination of dedicated hardware that performs some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) that performs other functions.

[0211] According to an embodiment of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the method for model training as described above is implemented.

[0212] According to an embodiment of the present disclosure, an electronic device is provided, including: a processor and a memory, storing a computer program, wherein when the computer program is executed by the processor, the method for model training as described above is implemented.

[0213] According to an embodiment of the present disclosure, a computer-readable storage medium storing a computer program may also be provided. When the program is run, the method for model training according to an embodiment of the present disclosure may be executed. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, for example, multimedia card, secure digital (SD) card and / or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and / or any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-temporary manner and provide the computer programs and any associated data, data files and / or data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In one example, the computer program and any associated data, data files and / or data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0214] According to the method, host and storage device for model training of the exemplary embodiments of the present disclosure, data offloading between the GPU and the storage device is used to solve the "memory wall" problem of the GPU, and data prefetching is used to speed up the training of the model, thereby achieving better training throughput; because the GPU and the storage device perform data transmission through direct communication, less CPU resources and main memory bandwidth can be used and robust performance can be provided; in addition, while the software implementation complexity is low, higher storage capacity and lower cost are provided for large model training.

[0215] While the present disclosure has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. A method for training a neural network model, the method comprising: The host device instructs a graphics processor GPU to load first intermediate data of a current layer of the model from a dynamic random access memory DRAM of a storage device into a memory of the GPU, wherein the current layer is a layer of the model that is performing calculations; and Based on the space capacity of the DRAM, the host device notifies the storage device to pre-fetch the second intermediate data of the layer to be calculated of the model from the NAND of the storage device to the DRAM of the storage device. The first intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer.

2. The method of claim 1, wherein: The pre-fetching of the second intermediate data is performed at least partially in parallel with the calculation for the current layer.

3. The method of claim 1, wherein: Notifying the storage device to pre-fetch the second intermediate data includes: Traversing the layers to be computed of the model as pre-fetch layers, and Based on the fact that the DRAM has remaining space and the prefetching of the second intermediate data has not been performed for the prefetching layer, the storage device is notified to prefetch the second intermediate data for the prefetching layer.

4. The method of claim 1, further comprising: In the forward propagation of the model, the GPU is instructed to unload third intermediate data generated by calculation of the layer of the model from the memory of the GPU to the NAND of the storage device.

5. The method of claim 4, further comprising: Before the model training, based on the memory amount of the GPU and the execution time of the forward propagation, obtaining information of unloading the third intermediate data to one or more layers of the NAND in the multiple layers of the model; Wherein, in the forward propagation of the model, instructing the GPU to unload the third intermediate data includes: Based on the information, in the forward propagation of the model, the GPU is instructed to unload the third intermediate data.

6. The method of claim 5, wherein: Based on the memory amount of the GPU and the execution time of the forward propagation, obtaining information of unloading the third intermediate data to one or more layers of the NAND in the multiple layers of the model, including: The information is derived based on the total amount of remaining intermediate data being less than or equal to the amount of memory of the GPU and the total offloading time being less than or equal to the execution time of the forward propagation, wherein the total amount of remaining intermediate data is based on an amount of fourth intermediate data of a layer other than a layer in which the third intermediate data is offloaded to the NAND among the multiple layers of the model, and The total unloading time is based on the unloading time of unloading the third intermediate data to the NAND layer.

7. The method of claim 1, wherein: The first intermediate data includes activation values, and the calculation for the current layer includes gradient calculation with respect to the activation values.

8. A method for training a neural network model, the method comprising: A storage device receives a notification of executing a data pre-fetch operation from a host, wherein the storage device includes: a dynamic random access memory DRAM and a NAND, and Based on the received notification, the storage device pre-fetches intermediate data of the to-be-calculated layer of the model from the NAND to the DRAM.

9. The method of claim 8, wherein: The method further comprises: Sending intermediate data of the current layer of the model to the memory of a graphics processor GPU, wherein the current layer of the model is the layer of the model that is performing calculations, wherein the intermediate data of the current layer is stored in the DRAM, and The intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer.

10. The method of claim 9, wherein: Pre-fetching of intermediate data of a layer of the model to be calculated is performed at least partially in parallel with calculation for the current layer.

11. The method of claim 9, wherein: The intermediate data of the layer to be calculated of the model is first intermediate data, and wherein the method further comprises: In the forward propagation of the model, second intermediate data is received, where the second intermediate data is data generated by calculation of a layer of the model and is unloaded from a memory of the GPU to be stored in the NAND.

12. The method of claim 11, wherein: Sending the intermediate data of the current layer includes: Based on the Compute Fast Link (CXL) protocol, the intermediate data of the current layer stored in the DRAM is sent to the memory of the GPU. The receiving of the second intermediate data includes: Based on the CXL protocol, second intermediate data generated by the calculation and unloaded from the memory of the GPU to be stored in the NAND is received.

13. The method of claim 9, wherein: The storage device also includes a memory-semantic solid-state drive MS SSD.

14. A host, comprising: Memory, which stores instructions; as well as A processor is configured to execute the instructions to perform the following operations: Instructing a graphics processor GPU to load first intermediate data of a current layer of a model from a dynamic random access memory DRAM of a storage device into a memory of the GPU, wherein the current layer is a layer of the model that is performing calculations; and Based on the space capacity of the DRAM, instruct the storage device to pre-fetch the second intermediate data of the layer to be calculated of the model from the NAND of the storage device to the DRAM, The first intermediate data of the current layer is used by the GPU in the back propagation of the model to perform calculations for the current layer.

15. A storage device comprising: Dynamic random access memory DRAM and NAND, the storage devices are configured as: Receive notification from the host to perform data prefetch operations, Based on the received notification, intermediate data of the to-be-calculated layer of the model is pre-fetched from the NAND to the DRAM.

16. A system using a storage device, comprising: Main processor; Memory; as well as the storage device, Wherein, the storage device is configured to execute the method for training a neural network model as described in any one of claims 8-13.

17. A host storage system, comprising: Host; as well as Storage devices, Wherein, the storage device is configured to execute the method for training a neural network model as described in any one of claims 8-13.

18. A data center system, comprising: Multiple application servers; as well as Multiple storage servers, Wherein, each storage server includes a storage device, wherein the storage device is configured to execute the method for training a neural network model as described in any one of claims 8-13.

19. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the method for training a neural network model as described in any one of claims 1 to 13 is implemented.

Images