Method and system for model training

Abstract

The invention provides a method and a system for model training. A training method of a deep learning model may include: grouping, by a graphics processing unit (GPU), parameters of each layer of the deep learning model; the GPU unloads the grouped parameters to a plurality of storage devices, and each storage device stores a group of parameters of each layer; and executing the training of the deep learning model in parallel by the GPU and the plurality of storage devices.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a method and system for model training. Background Art

[0002] With the development of artificial intelligence (AI) technology, models are becoming larger and larger. For example, large deep learning models (such as GPT-X (chatGPT)) have billions or even trillions of parameters generated by numerous layers and complex structures. During model training, large deep learning models require hundreds of graphics processing units (GPUs) to store model parameters such as model weights, gradients, optimizer parameters, and perform a large number of forward propagation (FP) and backward propagation (BP) operations. For example, a deep learning model with 1 trillion parameters requires at least 20TB of memory to store all the parameters required for training, which means that a large number of GPUs are required to ensure training. Therefore, GPU memory has become the main bottleneck for training large deep learning models. Summary of the invention

[0003] The present disclosure provides a method and system for model training to solve part or all of the above problems.

[0004] According to a first aspect of the present disclosure, a method for training a deep learning model is provided, comprising: a graphics processing unit (GPU) grouping parameters of each layer of the deep learning model; the GPU unloading the grouped parameters to multiple storage devices respectively, wherein each storage device stores a set of parameters for each layer; and the GPU and the multiple storage devices executing the training of the deep learning model in parallel.

[0005] The present disclosure realizes parallel operation by grouping the parameters of each layer of the model and unloading them to multiple storage devices, thereby avoiding the problem of increased training time cost caused by the need to wait for the parameters to be read from the storage device during model training in the GPU.

[0006] Optionally, the GPU and the multiple storage devices perform training of the deep learning model in parallel, including: when executing the forward propagation of the i-th layer of the deep learning model, the GPU loads the weights of the i+1-th layer of the deep learning model from the dynamic random access memory DRAM of the multiple storage devices into the memory of the GPU, where i is a positive integer; and the multiple storage devices pre-fetch the weights of the i+2-th layer of the deep learning model from the NAND of the multiple storage devices into the DRAM.

[0007] The present invention improves the reading speed between the GPU and the storage device and avoids training delays by reading weights in parallel and pre-fetching weights in parallel.

[0008] Optionally, the GPU and the multiple storage devices perform training of the deep learning model in parallel, including: when performing backward propagation of the deep learning model, the GPU groups the gradients of each layer and writes the grouped gradients into the NAND of the multiple storage devices respectively.

[0009] The present invention improves the writing speed between the GPU and the storage device by writing gradients in parallel, thereby avoiding training delays.

[0010] Optionally, the GPU and the multiple storage devices execute the training of the deep learning model in parallel, including: the field programmable gate arrays FPGAs in the multiple storage devices execute the weight update of the deep learning model in parallel.

[0011] The present invention speeds up the calculation and avoids training delays by updating weights in parallel.

[0012] Optionally, the field programmable gate arrays FPGA in the multiple storage devices execute the weight update of the deep learning model in parallel, including: in the FPGA in each storage device, the weight update of the deep learning model is executed in a data parallel manner and a pipeline parallel manner.

[0013] The present invention speeds up the calculation speed and avoids training delays by updating weights in parallel at multiple scales.

[0014] Optionally, the training method also includes: when executing the weight update of the deep learning model, the multiple storage devices store the currently updated weights in another device respectively; when the weight update is completed, the multiple storage devices delete the weights previously stored in the multiple storage devices and store the weights in the other device respectively in the multiple storage devices.

[0015] The present disclosure stores the updated weights in another storage space and retains the previous weights as a backup, which can not only avoid the problem that the first layer FP of the next iteration needs to wait for the weight update and pre-fetching in the storage device, but also avoid the problem that the parameters cannot be restored due to unexpected training.

[0016] Optionally, the plurality of storage devices include a memory-semantic solid-state drive (MS SSD).

[0017] The present disclosure adopts MS SSD, which can not only realize parameter pre-fetching, but also realize fast reading and writing. In addition, MS SSD introduces FPGA chip, which is suitable for offloading near data processing of parameters (such as weight update) to avoid frequent data copying.

[0018] According to a second aspect of the present disclosure, a deep learning model training system is provided, comprising: a controller; a graphics processor; and multiple storage devices; wherein the controller is configured to: control the graphics processor GPU to group the parameters of each layer of the deep learning model; control the GPU to unload the grouped parameters to multiple storage devices respectively, wherein each storage device stores a set of parameters for each layer; and control the GPU and the multiple storage devices to perform the training of the deep learning model in parallel.

[0019] Optionally, the controller is configured to: when executing forward propagation of the i-th layer of the deep learning model, control the GPU to load the weights of the i+1-th layer of the deep learning model from the dynamic random access memory DRAM of the multiple storage devices into the memory of the GPU, where i is a positive integer; and control the multiple storage devices to pre-fetch the weights of the i+2-th layer of the deep learning model from the NAND of the multiple storage devices into the DRAM.

[0020] Optionally, the controller is configured to: when executing the backward propagation of the deep learning model, control the GPU to group the gradients of each layer and write the grouped gradients into the NANDs of the multiple storage devices respectively.

[0021] Optionally, the controller is configured to: control the field programmable gate arrays FPGA in the multiple storage devices to execute weight updates of the deep learning model in parallel.

[0022] Optionally, the controller is configured to: in the FPGA in each storage device, control the FPGA to execute weight update of the deep learning model in a data parallel manner and a pipeline parallel manner.

[0023] Optionally, the controller is configured to: when executing the weight update of the deep learning model, control the multiple storage devices to store the currently updated weights in another device respectively; when the weight update is completed, control the multiple storage devices to delete the weights previously stored in the multiple storage devices and store the weights in the other device respectively in the multiple storage devices.

[0024] Optionally, the plurality of storage devices include a memory-semantic solid-state drive (MS SSD).

[0025] According to a third aspect of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the method as described above is implemented.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0028] Figure 1 Schematic diagram showing the DNN training process.

[0029] Figure 2 Schematic diagram showing GPU memory consumption during training of a deep neural network.

[0030] Figure 3 A schematic diagram showing the execution of model training in a GPU and a MS SSD according to an embodiment of the present disclosure.

[0031] Figure 4 A schematic diagram showing the structure of an MS SSD and the process of pre-fetching and updating weights according to an embodiment of the present disclosure.

[0032] Figure 5 is a schematic diagram of a layer-based grouping strategy according to an embodiment of the present disclosure.

[0033] Figure 6 An example of transmitting parameters from multiple MS SSDs to a GPU according to an embodiment of the present disclosure is shown.

[0034] Figure 7 A schematic diagram showing the parallel execution of model training by multiple MS SSDs according to an embodiment of the present disclosure.

[0035] Figure 8 A schematic diagram showing a data parallel mechanism and a pipeline parallel mechanism according to an embodiment of the present disclosure.

[0036] Fig. 9 It is a schematic diagram of a backup-based weight transmission strategy according to an embodiment of the present disclosure.

[0037] Fig.10 It is a flowchart of a method for model training according to an embodiment of the present disclosure.

[0038] Fig.11 It is a schematic diagram of the forward propagation process according to an embodiment of the present disclosure.

[0039] Fig.12 It is a schematic diagram of the back propagation process according to an embodiment of the present disclosure.

[0040] Fig.13 Detailed description of the invention is a block diagram of a system for model training according to an embodiment of the present invention.

[0041] Fig.14 is a schematic diagram of a system to which a storage device is applied according to an embodiment.

[0042] Fig.15 is a block diagram of a host storage system according to an embodiment.

[0043] Fig.16 is a diagram of a data center to which a storage device is applied according to an embodiment. DETAILED DESCRIPTION

[0044] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0045] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0046] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0047] Artificial intelligence models, such as Deep Neural Networks (DNN) models, consist of many interconnected layers in which samples are propagated. The training process of a deep neural network can be a process of repeated matrix calculations. Figure 1 A schematic diagram of the DNN training process is shown. Figure 1As shown, in the forward propagation, the activation function value of the current layer ( Figure 1 A L , L represents the Lth layer of the model) through the activation function value A of the previous layer L-1 With the weight of the current layer ( Figure 1 W L ) is multiplied and calculated layer by layer. In addition, the deviation between the expected value and the actual value is calculated ( Figure 1 b L In the backward propagation, the loss is propagated in a backward order to generate the gradient ( Figure 1 G L ) and use the momentum and variance of the optimizer ( Figure 1 m L and v L ) performs weight update.

[0048] from Figure 1 As can be seen in Figure 1, the model training process is orderly and predictable, which provides an opportunity for memory offloading. The training parameters in the GPU (such as activation function values, weights, gradients, momentum, and variance) can be offloaded to other places (such as DRAM or storage), and some of them can be transferred back when the GPU calculation needs them. The orderly and predictable process is conducive to prefetching and makes the GPU insensitive to memory offloading.

[0049] Figure 2 Schematic diagram showing GPU memory consumption during training of a deep neural network.

[0050] In forward propagation (FP), weights and biases ( Figure 2 W and b in, such as W 1 represents the weight of the first layer of the model, b 1 represents the bias of the first layer of the model) and the activation function value ( Figure 2 A in, such as A 1 The activation function value representing the first layer of the model is stored as a 16-bit floating point number, ensuring high throughput of the GPU core. In the present disclosure, weights and biases of floating point precision 16 can be represented as weights 16, and activation function values ​​of floating point precision 16 can be represented as activation function values ​​16.

[0051] In Backward Propagation (BP), the gradient of floating point precision 16 is calculated by the derivative of the activation function value 16 ( Figure 2 G in, such as G 1 represents the gradient of the first layer of the model). In the present disclosure, the gradient of floating point precision 16 can be represented as gradient 16. Combined with gradient 16, the momentum and variance of the optimizer ( Figure 2 m and v in, such as m 1 and v 1 represents the momentum and variance of the first layer of the model) to update the weights ( Figure 2 W* in ), which are stored as 32-bit floating point numbers. In the present disclosure, momentum, variance, and weight of floating point precision 32 are represented as momentum 32, variance 32, and weight 32.

[0052] Generally speaking, a neural network model during training includes 16 weights, 16 activation function values, 16 gradients, 32 momentums, 32 variances, and 32 weights, which consumes a lot of GPU memory.

[0053] As the size of large deep learning models increases, the ratio of model size to AI hardware (such as GPU) memory is as high as 100 times or even 1000 times, which means that the training cost of large deep learning models is huge and model reconstruction is complex and difficult. To solve this problem, there are two main offloading methods:

[0054] 1) ZeRO-offload (ZeRO-Offload is a method to reduce the GPU memory usage of deep learning models during training by offloading data and computation from the GPU to the CPU).

[0055] Specifically, the weights 32 and the optimizer parameters are offloaded to the host memory, while the activation function values ​​are not offloaded because there is an activation checkpoint technology to reduce the memory occupied by most activation function values.

[0056] Weight updates and float2half calculations are offloaded to the CPU so that weight updates and float2half calculations are performed by the CPU. Calculations for both forward and backward propagation are performed by the GPU.

[0057] 2) ZeRO-Cache (ZeRO-Cache is a method to reduce GPU memory usage during training of deep learning models by offloading data from the GPU to storage devices)

[0058] Specifically, the weights 32 and the parameters of the optimizer are unloaded to the SSD, and the weights 16 and the gradients 16 are unloaded to the host memory (such as RAM). The CPU performs calculations for updating weights by pre-fetching relevant weights from the SSD. The GPU performs calculations for forward propagation and backward propagation using the weights 16 pre-fetched from RAM.

[0059] There are several problems with the above uninstallation method:

[0060] 1. RAM and CPU costs are high. Large deep learning models have a large number of parameters. Existing solutions offload parameters to host memory (RAM) and offload calculations on weight updates to the CPU. However, large deep learning models (such as GPT-4) can reach tens or even hundreds of TB in size during training, which requires a lot of RAM and is very expensive. Offloading calculations to the CPU may also leave the CPU without enough cores to perform common operations in training, such as data processing and data enhancement, so the CPU will become a new bottleneck.

[0061] 2. High communication and replication costs. ZeRO-Cache offloads some model parameters to RAM and SSD, but the communication and data replication between SSD, RAM, and GPU are very complicated, and data read and write IO operations will become a new bottleneck.

[0062] 3. The parameters are highly volatile. Since the parameters of the above uninstallation method will be inconsistent during the update process, when an accident occurs, it is impossible to restore the latest parameters during training.

[0063] In order to solve the GPU memory problem and overcome the shortcomings of the above solutions, the present disclosure proposes a method for model training, which offloads the components of deep learning model training from the GPU to a memory semantic solid-state drive (MSSSD) to reduce GPU memory consumption, and because the parameters of the deep learning model are offloaded to the MS SSD instead of the RAM, the calculation of weight update is performed by the FPGA of the MS SSD instead of the CPU, and the price of the MS SSD is lower. Therefore, the cost of the MS SSD is much lower than that of the RAM and the CPU, thereby reducing the cost of model training.

[0064] The model disclosed in the present invention may be a deep neural network (DNN) model, or other models including multiple layers, for example, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a generative adversarial network (GAN) model, a long short-term memory network (LSTM) model, a residual network (ResNet, such as ResNet-1922) model, an attention mechanism (AttentionMechanism) model, a Transformer model, a GPT model, a visual geometry group (VGG) model, an OverFeat model, a densely connected network (DenseNet, such as DenseNet-1001) model, a GoogLeNet model, and an AlexNet model, etc. However, the present invention is not limited thereto, and the model may also be other types of artificial intelligence models.

[0065] The components of GPU training deep learning models may involve calculations (such as forward propagation FP, backward propagation BP, weight updates, and conversion of float32 to float16) and parameters (such as weight 16, activation function value 16, gradient 16, optimizer parameters (momentum 32, variance 32, and weight 32).

[0066] Figure 3 A schematic diagram showing the execution of model training in a GPU and a MS SSD according to an embodiment of the present disclosure.

[0067] Reference Figure 3 , considering that the optimizer parameters account for a large proportion and are only used for weight update, they are not needed in FP and BP, so the optimizer parameters can be unloaded. In addition, the weight 16 of a certain layer is only needed when training to this layer, so the weight 16 can also be unloaded. The above unloaded parameters are used for weight update. Compared with FP and BP calculations, the weight update with lower complexity and the simple calculation from float32 to float16 can also be unloaded to the Field Programmable Gate Array (FPGA) in the MS SSD.

[0068] Figure 4 A schematic diagram showing the structure of an MS SSD and the process of pre-fetching and updating weights according to an embodiment of the present disclosure.

[0069] GPU can communicate with MS SSD via Compute Express Link (CXL) protocol. The high-performance mode of MS SSD includes the following functions: 1) Data pre-fetch operation (hardware cache): pre-fetch is to pre-load data from NAND into DRAM; 2) Support dual-mode access: NVMe read / write through CXL.io, and logical block address (LBA) memory read through CXL.mem.

[0070] Since the training process of deep learning models is orderly and predictable, the pre-fetch interface of MS SSD can pre-fetch the parameters of the next training into DRAM in advance, and quickly load the parameters into the GPU through CXL.mem. Figure 4 , when executing the forward propagation of the i-th layer (Layer i) of the deep learning model, the GPU loads the weight of the i+1-th layer (Layer i+1) of the deep learning model from the dynamic random access memory DRAM of the MS SSD to the memory of the GPU through CXL.mem, where i is a positive integer, and the weight of the i+2-th layer (Layer i+2) of the deep learning model is pre-fetched from the NAND to the DRAM by the MS SSD.

[0071] CXL.mem reads from DRAM at up to 25GB / s, and CXL.io writes faster as well.

[0072] The FPGA chip introduced in the MS SSD can be used for near data processing and calculation, and is suitable for offloading near data processing of parameters (such as weight update) to avoid frequent data replication (for example, if the weight update is still in the GPU, the optimizer parameters need to be frequently transmitted between the GPU and the MS SSD, resulting in an IO bottleneck). Therefore, the present disclosure offloads the parameters and calculations of the deep learning model to the MS SSD.

[0073] The present disclosure takes into account that if the weights 16 and gradients 16 are not transferred in time, the model training may be delayed. Therefore, a layer-based grouping unloading strategy is proposed, that is, the parameters of each layer of the model are grouped and unloaded to multiple MS SSDs for parallel transmission to solve the IO bottleneck between the GPU and the MS SSD.

[0074] Figure 5 is a schematic diagram of a layer-based grouping strategy according to an embodiment of the present disclosure.

[0075] Since the IO between GPU and MS SSD is slow, the training in GPU may wait for parameters to be read from MS SSD. This training delay will increase the time cost. To overcome this problem, the present disclosure designs this layer-based grouping strategy.

[0076] Reference Figure 5 , the parameters of each layer of the deep learning model are evenly divided into multiple groups of parameters, such as L1-G1 represents the first group of parameters of the first layer of the model, L1-G2 represents the second group of parameters of the first layer of the model, LN-G1 represents the first group of parameters of the Nth layer of the model, and so on.

[0077] The MS SSD pool is built to store parameters. The grouped parameters of each layer can be unloaded to each MS SSD in the MSSSD pool. For example, the first set of parameters of the first layer of the model, the first set of parameters of the second layer, ..., the first set of parameters of the Nth layer are unloaded to the NAND of the first MS SSD, and the second set of parameters of the first layer of the model, the second set of parameters of the second layer, ..., the second set of parameters of the Nth layer are unloaded to the NAND of the second MS SSD, and so on. In this way, the above-mentioned IO operations (i.e., the transmission of weights 16 and gradients 16) and weight update operations can be performed in parallel, thereby avoiding training delays.

[0078] Figure 6 An example of transmitting parameters from multiple MS SSDs to a GPU according to an embodiment of the present disclosure is shown.

[0079] Assume that the size of weight 16 of the Kth layer is 80M, the FP of the K-1th layer is about 1ms, and the speed of CXL.mem is 25G / s=25M / ms. In the conventional FP case, only 25M (25M / ms*1ms) parameters of the Kth layer can be read in the FP of the K-1th layer, which delays the FP of the Kth layer. According to the embodiment of the present disclosure, referring to Figure 5 , assuming that there are four MS SSDs for parallel IO, the weight 16 of the Kth layer can be fully read in the FP of the K-1th layer (25M*4=100M>80M), and there is no training delay.

[0080] Figure 7 A schematic diagram showing a model training performed in parallel by multiple MS SSDs according to an embodiment of the present disclosure is shown. Figure 7 In the figure, next in different pattern boxes represents different parameters of the next layer.

[0081] Reference Figure 7 ,In each MS SSD, the weights of the next layer are prefetched into DRAM before the previous layer is trained to achieve parallel prefetching.

[0082] During the training of the previous layer, the weights of the next layer in the DRAM of multiple MS SSDs are read into the GPU in parallel through CXL.mem to achieve parallel reading.

[0083] If in the BP process, the gradients after BP will be grouped and written to the NAND of each MS SSD through CXL.io to achieve parallel writing.

[0084] The weight updates of multiple parameter groups (such as parameter group 1, parameter group 2, etc.) are performed in parallel on multiple FPGAs in multiple MS SSDs to achieve parallel weight updates. This is the first parallelization of weight updates, which speeds up the calculation.

[0085] Considering that the calculation of updating weights in the FPGA of the MS SSD is slower than that in the GPU, untimely weight updates will delay training. In this paper, a multi-scale parallel weight update method is designed, that is, the FPGA in each MS SSD uses a combination of data parallelism and pipeline parallelism to accelerate the weight update calculation.

[0086] By adopting parallel pre-fetch / read / write / near data calculation in MS SSD, the calculation speed of FPGA in MS SSD can be accelerated.

[0087] Two computing mechanisms are provided in each FPGA, namely, a data parallel mechanism and a pipeline parallel mechanism. Figure 8 A schematic diagram showing a data parallel mechanism and a pipeline parallel mechanism according to an embodiment of the present disclosure.

[0088] Reference Figure 8 ,In data parallel mechanism, data is divided into multiple groups / blocks and the same operation is performed by the corresponding ,processing units.,In pipeline parallel mechanism, the operation is divided into multiple steps and the steps are ,performed in parallel by the operation units.

[0089] According to an embodiment of the present disclosure, weight updates are performed in parallel at multiple scales. Specifically, in data parallel operations, all model parameters are divided into multiple blocks, and then the ADAM optimization algorithm is used in each block to perform weight update calculations. In pipeline parallel operations, in the calculation of each block, the calculation of updated weights is divided into multiple steps. Multiple steps can be calculated in parallel by multiple operation units, and then combined to complete the weight update. The above-mentioned data parallel operations and pipeline parallel operations are performed in each FPGA of each MS SSD, thereby realizing parallel weight updates of multiple MS SSDs. The calculation of the ADAM algorithm in the MS SSD FPGA is accelerated by using a multi-scale parallel weight update method of data parallelism and pipeline parallelism by multiple MS SSDs.

[0090] With this approach, the offloaded weight update computations will be greatly accelerated, so this computation offloading will not hinder training, and by computing near the data, data transfer bottlenecks (such as frequent transmission of optimizer parameters) are avoided.

[0091] According to the present disclosure, considering the waiting time for weight update (especially offloading to MSSSD for execution) between the previous BP and the next FP, the present disclosure designs a backup-based weight transmission strategy, which backs up the previous version of the weight for the next FP to avoid waiting for weight update.

[0092] Fig. 9 Schematic diagram of a backup-based weight transfer strategy according to an embodiment of the present disclosure. This strategy is used when the influence of gradients on model training is negligible.

[0093] Reference Fig. 9 , this strategy stores the new updated weights in another storage space instead of overwriting them, so that the old weights can be kept as a backup. The old weights can be read during iterative training without waiting for the weight update to complete. When the weight update is completed, the new weights replace the old weights, that is, the new weights become the old backup weights. This strategy is suitable for the common situation where the weight difference between two training iterations is small.

[0094] In addition, the optimizer parameters are offloaded to the NAND of the MS SSD, and they are non-volatile, so this strategy can be used for accident recovery. During the weight update process, the new weights are not fully obtained, so the backup weights can be used for accidental recovery. The backup weights are always consistent because they can only be replaced after all new weights are obtained. If the training stops unexpectedly and training recovery is required, the backup weights will be used to initialize the model. The backup weights are read from the MS SSD to the GPU and the model training continues.

[0095] This strategy stores two copies of the weights, allowing for immediate recovery of a failed model if something unexpected happens during training.

[0096] Fig.10 It is a flowchart of a method for model training according to an embodiment of the present disclosure.

[0097] Reference Fig.10 In step S1001, the graphics processor GPU groups the parameters of each layer of the deep learning model.

[0098] In step S1002, the GPU unloads the grouped parameters to multiple storage devices, where each storage device stores a set of parameters for each layer. The multiple storage devices may include a memory-semantic solid-state drive MS SSD. By unloading most of the parameters in the deep learning model training, the GPU memory consumption can be greatly reduced.

[0099] In step S1003, the GPU and multiple storage devices perform training of the deep learning model in parallel.

[0100] When performing forward propagation of the i-th layer of the deep learning model, the GPU may load the weights of the i+1-th layer of the deep learning model from the dynamic random access memory DRAM of multiple storage devices into the memory of the GPU, where i is a positive integer. The weights of the i+2-th layer of the deep learning model may be pre-fetched from the NAND of multiple storage devices into the DRAM by multiple storage devices.

[0101] Fig.11 It is a schematic diagram of the forward propagation process according to an embodiment of the present disclosure.

[0102] Reference Fig.11 After the model is initialized and at the beginning of model training, the model parameters can be divided using a layer-based grouping offloading strategy and the divided parameters can be unloaded and stored in multiple MS SSDs. The GPU and multiple MS SSDs perform model training in parallel.

[0103] The GPU may perform forward propagation operations layer by layer. After the GPU performs forward propagation of the i-th layer, it may determine whether the i-th layer is the last layer. When the i-th layer is not the last layer, the GPU may continue to perform forward propagation of the next layer (i.e., the i+1-th layer). When the i-th layer is the last layer, the GPU may perform backward propagation.

[0104] MS SSD can determine whether the i-th layer is the last layer. If it is the last layer, MS SSD stops prefetching. If it is not the last layer, MS SSD can determine whether the i-th layer is the second last layer and prefetch the weights of the next layer (i.e., the i+1th layer) from DRAM to the GPU for GPU to perform feedforward propagation of the next layer. If it is the second last layer, MS SSD stops prefetching. If it is not the second last layer, MS SSD prefetches the weights of the i+2th layer from NAND to DRAM.

[0105] When performing the back propagation of the deep learning model, the GPU can group the gradients of each layer and write the grouped gradients into the NAND of multiple storage devices respectively.

[0106] Fig.12 It is a schematic diagram of the back propagation process according to an embodiment of the present disclosure.

[0107] Reference Fig.12 , back-propagation operations can be performed in the GPU. Gradients can be calculated layer by layer and written to the NAND of each MS SSD using a layer-based group offload strategy for weight updates.

[0108] In each MS SSD, weight updates can be performed by FPGA.

[0109] Determine whether the backpropagation is completed. If completed, determine whether the weight update is completed. If not, continue to calculate the gradient.

[0110] If the weight update is completed, the previously stored old weights can be deleted, and the updated new weights can be stored in NAND as a backup for forward propagation. If the weight update is not completed, the old weights can be used to perform forward propagation.

[0111] The weight updates of the deep learning model can be performed in parallel by field programmable gate arrays (FPGAs) in multiple storage devices.

[0112] The present disclosure performs parallel pre-fetching / reading / writing / weight updating based on multiple MS SSDs, thereby avoiding IO bottlenecks and training delays.

[0113] In the FPGA in each storage device, the weight update of the deep learning model can be performed in a data parallel manner and a pipeline parallel manner.

[0114] The present invention offloads the calculation of near data (weight update) to MS SSD to avoid data transmission bottleneck, and adopts multi-scale parallel weight update to accelerate the near data calculation and keep up with the GPU training speed.

[0115] When executing the weight update of the deep learning model, multiple storage devices can store the currently updated weights in another device respectively. The other device can be a storage space in the MS SSD that is different from NAND, or other storage devices that can be connected to the MS SSD. When the weight update is completed, the multiple storage devices can delete the weights previously stored in the multiple storage devices and store the weights in another device in the multiple storage devices respectively.

[0116] The present disclosure adopts backup-based weight transfer to avoid waiting for weight update before the next FP, and weight backup can also be applied to training crash recovery.

[0117] Fig.13 Detailed description of the invention is a block diagram of a system for model training according to an embodiment of the present invention.

[0118] Reference Fig.13 , the system 1300 for model training may include a controller 1301, a GPU 1302, and multiple storage devices 1303.

[0119] The controller 1301 may be implemented by a host. The storage device 1303 may include a dynamic random access memory DRAM and a flash memory NAND, and the training of the model is performed in the GPU. The storage device 1303 may be a memory-semantics solid state drive (MS SSD).

[0120] In the following example embodiments, the storage device is described by taking a MS SSD as an example, but it should be understood that the present disclosure is not limited thereto. For example, the storage device may also be any storage device that provides a hardware cache (eg, DRAM).

[0121] The controller 1301 can control the GPU 1302 to group the parameters of each layer of the deep learning model, control the GPU 1302 to unload the grouped parameters to multiple storage devices 1303 respectively, wherein each storage device 1303 stores a set of parameters for each layer, and control the GPU 1302 and the multiple storage devices 1303 to perform the training of the deep learning model in parallel.

[0122] When executing forward propagation of the i-th layer of the deep learning model, the controller 1301 can control the GPU 1302 to load the weights of the i+1-th layer of the deep learning model from the dynamic random access memory DRAM of multiple storage devices 1303 into the memory of the GPU 1302, where i is a positive integer, and control the multiple storage devices 1303 to pre-fetch the weights of the i+2-th layer of the deep learning model from the NAND of multiple storage devices 1303 into the DRAM.

[0123] When executing the backward propagation of the deep learning model, the controller 1301 can control the GPU 1302 to group the gradients of each layer and write the grouped gradients into the NAND of multiple storage devices 1303 respectively.

[0124] The controller 1301 can control the field programmable gate arrays FPGA in multiple storage devices 1303 to execute weight updates of the deep learning model in parallel.

[0125] In the FPGA in each storage device 1303, the controller 1301 can control the FPGA to perform weight updates of the deep learning model using data parallelism and pipeline parallelism.

[0126] When executing the weight update of the deep learning model, the controller 1301 can control multiple storage devices 1303 to store the currently updated weights in another device respectively. When the weight update is completed, the controller 1301 can control the multiple storage devices 1303 to delete the weights previously stored in the multiple storage devices 1303 and store the weights in the other device respectively in the multiple storage devices 1303.

[0127] Large deep learning models are widely used in fields such as semantic analysis and computer vision, which consume a lot of hardware memory. The strategies and algorithms disclosed in this disclosure can be directly used to offload the parameters of large deep learning models to MS SSD. MS SSD can be used to reduce GPU memory requirements and training costs.

[0128] Fig.14 is a schematic diagram of a system 1000 to which a storage device is applied according to an embodiment.

[0129] Fig.14 The system 1000 may be, for example, a mobile system such as a portable communication terminal (e.g., a mobile phone), a smart phone, a tablet personal computer (PC), a wearable device, a healthcare device, or an Internet of Things (IOT) device. However, Fig.14 The system 1000 is not limited thereto and may be, for example, a PC, a laptop computer, a server, a media player, or an automotive device (eg, a navigation device).

[0130] Reference Fig.14, the system 1000 may include a main processor 1100, a memory (e.g., 1200a and 1200b), and a storage device (e.g., 1300a and 1300b). In addition, the system 1000 may include at least one of an image capture device 1410, a user input device 1420, a sensor 1430, a communication device 1440, a display 1450, a speaker 1460, a power supply device 1470, and a connection interface 1480.

[0131] The main processor 1100 may control all operations of the system 1000, including, for example, operations of other components included in the system 1000. The main processor 1100 may be implemented as, for example, a general-purpose processor, a dedicated processor, a graphics processor, an application processor, or the like.

[0132] The main processor 1100 may include at least one central processing unit (CPU) core 1110 and a controller 1120 for controlling memories 1200a and 1200b and / or storage devices 1300a and 1300b. In some exemplary embodiments, the main processor 1100 may further include an accelerator 1130, which is a dedicated circuit for high-speed data operations such as, for example, artificial intelligence (AI) data operations. The accelerator 1130 may include, for example, a graphics processing unit (GPU), a neural processing unit (NPU), and / or a data processing unit (DPU), etc., and may be implemented as a chip physically separated from other components of the main processor 1100.

[0133] The memories 1200a and 1200b may be used as main storage devices of the system 1000. Although the memories 1200a and 1200b may respectively include volatile memories such as, for example, static random access memory (SRAM) and / or dynamic random access memory (DRAM), etc., according to an embodiment, the memories 1200a and 1200b may respectively include non-volatile memories such as, for example, flash memory, phase change random access memory (PRAM) and / or resistive random access memory (RRAM), etc. The memories 1200a and 1200b may be implemented in the same package as the main processor 1100.

[0134] The storage devices 1300a and 1300b may be used as non-volatile storage devices, which are configured to store data regardless of whether they are powered, and may have a larger storage capacity than the memories 1200a and 1200b. The storage devices 1300a and 1300b may include memory controllers (STRG CTRL) 1310a and 1310b and non-volatile memories (NVM) 1320a and 1320b, respectively, which are configured to store data under the control of the memory controllers 1310a and 1310b. Although the NVMs 1320a and 1320b may include V-NAND flash memories having a two-dimensional (2D) or three-dimensional (3D) structure, the NVMs 1320a and 1320b may include other types of NVMs, such as, for example, PRAM and / or RRAM, etc.

[0135] The storage devices 1300a and 1300b may be physically separated from the main processor 1100 and included in the system 1000, or may be implemented in the same package as the main processor 1100. The storage devices 1300a and 1300b may be solid state devices (SSDs) or memory cards, and may be removably combined with other components of the system 100 through interfaces such as the connection interface 1480 described further later. The storage devices 1300a and 1300b may be devices to which standard protocols such as, for example, universal flash memory (UFS), embedded multimedia card (eMMC), or NVMe are applied. However, the storage devices 1300a and 1300b are not limited thereto.

[0136] The image capture device 1410 can capture still images or moving images. The image capture device 1410 may include, for example, a camera, a camcorder, and / or a webcam.

[0137] The user input device 1420 may receive various types of data input by a user of the system 1000 and may include, for example, a touch pad, a keypad, a keyboard, a mouse, a microphone, and the like.

[0138] The sensor 1430 may detect various types of physical quantities that may be obtained from the outside of the system 1000 and convert the detected physical quantities into electrical signals. The sensor 1430 may include, for example, a temperature sensor, a pressure sensor, an illumination sensor, a position sensor, an acceleration sensor, a biosensor, and / or a gyro sensor.

[0139] The communication device 1440 may transmit and receive signals between other devices outside the system 1000 according to various communication protocols. The communication device 1440 may include, for example, an antenna, a transceiver, or a modem.

[0140] Display 1450 and speaker 1460 may be used as output devices configured to output visual information and auditory information, respectively, to a user of system 1000 .

[0141] The power supply device 1470 may appropriately convert power supplied from a battery embedded in the system 1000 and / or an external power source, and supply the converted power to each component of the system 1000 .

[0142] The connection interface 1480 may provide a connection between the system 1000 and an external device that is connected to the system 1000 and is capable of transmitting and receiving data to and from the system 1000. The connection interface 1480 may be implemented by using various interface schemes, such as, for example, Advanced Technology Attachment (ATA), Serial ATA (SATA), External Serial ATA (e-SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), Peripheral Component Interconnect (PCI), PCI express (PCIe), NVMe, IEEE 1394, Universal Serial Bus (USB) interface, Secure Digital (SD) card interface, Multimedia Card (MMC) interface, Embedded Multimedia Card (eMMC) interface, UFS interface, Embedded UFS (eUFS) interface, Compact Flash (CF) card interface, and the like.

[0143] According to an embodiment of the present disclosure, a system (e.g., 1000) applying a storage device is provided, including: a main processor (e.g., 1100); a memory (e.g., 1200a and 1200b); and a storage device (e.g., 1300a and 1300b), wherein the storage device is configured to execute the method for model training as described above.

[0144] Fig.15 is a block diagram of a host storage system 10 according to an embodiment.

[0145] The host storage system 10 may include a host 100 and a storage device 200. The storage device 200 may include a memory controller 210 and an NVM 220. According to an embodiment, the host 100 may include a host controller 110 and a host memory 120. The host memory 120 may be used as a buffer memory configured to temporarily store data to be transmitted to or received from the storage device 200.

[0146] The storage device 200 may include a storage medium configured to store data in response to a request from the host 100. As an example, the storage device 200 may include at least one of an SSD, an embedded memory, and a removable external memory. When the storage device 200 is an SSD, the storage device 200 may be a device that complies with the NVMe standard. When the storage device 200 is an embedded memory or an external memory, the storage device 200 may be a device that complies with the UFS standard or the eMMC standard. Both the host 100 and the storage device 200 may generate a packet according to the adopted standard protocol and may send the packet.

[0147] When the NVM 220 of the storage device 200 includes a flash memory, the flash memory may include a 2D NAND memory array or a 3D (or vertical) NAND (VNAND) memory array. As another example, the storage device 200 may include various other types of NVM. For example, the storage device 200 may include a magnetic random access memory (MRAM), a spin transfer torque MRAM, a conductive bridge RAM (CBRAM), a ferroelectric RAM (FRAM), a PRAM, a RRAM, and various other types of memory.

[0148] According to an exemplary embodiment, the host controller 110 and the host memory 120 may be implemented as separate semiconductor chips. Alternatively, in some exemplary embodiments, the host controller 110 and the host memory 120 may be integrated in the same semiconductor chip. As an example, the host controller 110 may be any one of a plurality of devices included in an application processor (AP). The AP may be implemented as, for example, a system on chip (SoC). In addition, the host memory 120 may be an embedded memory included in the AP or a memory device external to the AP.

[0149] The host controller 110 may manage an operation of storing data (eg, write data) of the buffer area of ​​the host memory 120 in the NVM 220 or an operation of storing data (eg, read data) of the NVM 220 in the buffer area.

[0150] The memory controller 210 may include a host interface 211, a memory interface 212, and a CPU 213, a flash translation layer (FTL), a packet manager 215, a buffer memory 216, an error correction code (ECC) engine 217, and an advanced encryption standard (AES) engine 218. The memory controller 210 may further include a working memory in which the FTL 214 is loaded. The CPU 213 may execute the FTL 214 to control data write and read operations on the NVM 220.

[0151] The host interface 211 may transmit and receive packets to and from the host 100. The packets transmitted from the host 100 to the host interface 211 may include commands or data to be written to the NVM 220, etc. The packets transmitted from the host interface 211 to the host 100 may include responses to commands or data read from the NVM 220, etc. The memory interface 212 may transmit data to be written to the NVM 220 to the NVM 220 or receive data read from the NVM 220. The memory interface 212 may be configured to comply with a standard protocol such as, for example, Toggle or Open NAND Flash Interface (ONFI).

[0152] The FTL 214 may perform various functions, such as, for example, an address mapping operation, a wear leveling operation, and a garbage collection operation. The address mapping operation may be an operation of converting a logical address received from the host 100 into a physical address for actually storing data in the NVM 220. The wear leveling operation may be a technique for preventing or reducing excessive degradation of a specific block by allowing the blocks of the NVM 220 to be used evenly. As an example, the wear leveling operation may be implemented by using a firmware technique that balances the erase counts of physical blocks. The garbage collection operation may be a technique for ensuring available capacity in the NVM 220 by erasing an existing block after copying valid data of the existing block to a new block.

[0153] The packet manager 215 may generate a packet according to a protocol agreed with an interface of the host 100, or parse various types of information from a packet received from the host 100. In addition, the buffer memory 216 may temporarily store data to be written to or read from the NVM 220. Although the buffer memory 216 may be a component included in the memory controller 210, in an embodiment, the buffer memory 216 may be provided outside the memory controller 210.

[0154] The ECC engine 217 may perform error detection and correction operations on the read data read from the NVM 220. For example, the ECC engine 217 may generate parity bits for write data to be written to the NVM 220, and the generated parity bits may be stored together with the write data in the NVM 220. During the reading of data from the NVM 220, the ECC engine 217 may correct errors in the read data by using the read data and the parity bits read from the NVM 220, and output the error-corrected read data.

[0155] The AES engine 218 may perform at least one of an encryption operation and a decryption operation on data input to the memory controller 210 by using a symmetric key algorithm.

[0156] According to an embodiment of the present disclosure, a host storage system (e.g., 10) is provided, comprising: a host (e.g., 100); and a storage device (200), wherein the storage device is configured to execute the method for model training as described above.

[0157] Fig.16 is a diagram of a data center 3000 to which a storage device is applied according to an exemplary embodiment.

[0158] Platform part - server (application / storage)

[0159] Reference Fig.16 , the data center 3000 may be a facility that collects various types of data and provides services, and may be referred to as a data storage center. The data center 3000 may be a system for operating a search engine and a database, and may be a computing system used by a company (such as a bank) or a government agency. The data center 3000 may include application servers 3100 to 3100n and storage servers 3200 to 3200m, where n and m are positive integers. According to an exemplary embodiment, the number of application servers 3100 to 3100n and the number of storage servers 3200 to 3200m may be selected differently. The number of application servers 3100 to 3100n and the number of storage servers 3200 to 3200m may be different from each other.

[0160] The application server 3100 or the storage server 3200 may include processors 3110 and 3210 and at least one of memories 3120 and 3220, at least one of switches 3130 to 3130n, at least one of network interface cards (NICs) 3140 to 3140n and 3240 to 3240m, at least one of DRAMs 3253 to 3253m, and at least one of controllers 3251 to 3251m. Now, the storage server 3200 will be described as an example. The processor 3210 may control all operations of the storage server 3200, access the memory 3220, and execute instructions and / or data loaded into the memory 3220. The memory 3220 may be, for example, a double data rate synchronous DRAM (DDR SDRAM), a high bandwidth memory (HBM), a hybrid memory cube (HMC), a dual in-line memory module (DIMM), an Optane DIMM, or a non-volatile DIMM (NVMDIMM). In some exemplary embodiments, the number of processors 3210 and memories 3220 included in the storage server 3200 may be selected differently. In an exemplary embodiment, the processor 3210 and the memory 3220 may provide a processor-memory pair. In an exemplary embodiment, the number of processors 3210 and the number of memories 3220 may be different from each other. The processor 3210 may include a single-core processor or a multi-core processor. The above description of the storage server 3200 may be similarly applied to the application server 3100. In some embodiments, the application server 3100 may not include a storage device 3150. The storage server 3200 may include at least one storage device 3250. According to an embodiment, the number of storage devices 3250 included in the storage server 3200 may be selected differently.

[0161] Platform part - Network

[0162] The application servers 3100 to 3100n can communicate with the storage servers 3200 to 3200m through the network 3300. The network 3300 can be implemented by using a fiber channel (FC) or Ethernet. In this case, FC can be a medium for relatively high-speed data transmission, and an optical switch with high performance and high availability can be used. According to the access method of the network 3300, the storage servers 3200 to 3200m can be set to file storage, block storage, or object storage.

[0163] In one embodiment, the network 3300 may be a network dedicated to storage, such as a storage area network (SAN). For example, the SAN may be an FC-SAN, which uses an FC network and is implemented according to the FC protocol (FCP). As another example, the SAN may be an Internet Protocol (IP)-SAN, which uses a transmission control protocol (TCP) / IP network and is implemented according to a SCSI or Internet SCSI (iSCSI) protocol over TCP / IP. In one embodiment, the network 3300 may be a general network, such as a TCP / IP network. For example, the network 3300 may be implemented according to protocols such as FC over Ethernet (FCoE), network attached storage (NAS), and NVMe over fabric (NVMe-oF).

[0164] Hereinafter, the application server 3100 and the storage server 3200 will be mainly described. The description of the application server 3100 may be applied to another application server 3100n, and the description of the storage server 3200 may be applied to another storage server 3200m.

[0165] The application server 3100 may store data requested to be stored by a user or client in one of the storage servers 3200 to 3200m through the network 3300. In addition, the application server 3100 may obtain data requested to be read by a user or client from one of the storage servers 3200 to 3200m through the network 3300. For example, the application server 3100 may be implemented as a network server or a database management system (DBMS).

[0166] The application server 3100 may access a memory 3120n or a storage device 3150n included in another application server 3100n through the network 3300. Alternatively, the application server 3100 may access a memory 3220 to 3220m or a storage device 3250 to 3250m included in the storage servers 3200 to 3200m through the network 3300. Therefore, the application server 3100 may perform various operations on the data stored in the application servers 3100 to 3100n and / or the storage servers 3200 to 3200m. For example, the application server 3100 may execute an instruction for moving or copying data between the application servers 3100 to 3100n and / or the storage servers 3200 to 3200m. In this case, data may be moved from the storage devices 3250 to 3250m of the storage servers 3200 to 3200m through the storage 3220 to 3220m of the storage servers 3200 to 3200m or directly to the storage 3120 to 3120n of the application servers 3100 to 3100n. The data moved through the network 3300 may be data encrypted for security or privacy.

[0167] Organic Relationship-Interface Structure / Type

[0168] The storage server 3200 will now be described as an example. The interface 3254 can provide a physical connection between the processor 3210 and the controller 3251 and a physical connection between the network interface card (NIC) 3240 and the controller 3251. For example, the interface 3254 can be implemented using a direct attached storage (DAS) solution, where the storage device 3250 is directly connected to a dedicated cable. For example, the interface 3254 can be implemented using various interface solutions, such as ATA, SATA, e-SATA, SCSI, SAS, PCI, PCIe, NVMe, IEEE 1394, USB interface, SD card interface, MMC interface, eMMC interface, UFS interface, eUFS interface, and CF card interface.

[0169] The storage server 3200 may further include a switch 3230 and a network interconnect (NIC) 3240. The switch 3230 may selectively connect the processor 3210 to the storage device 3250 under the control of the processor 3210, or selectively connect the NIC 3240 to the storage device 3250.

[0170] In one embodiment, the NIC 3240 may include a network interface card and a network adapter. The NIC 3240 may be connected to the network 3300 through, for example, a wired interface, a wireless interface, a Bluetooth interface, or an optical interface. The NIC 3240 may include an internal memory, a digital signal processor (DSP), and a host bus interface, and is connected to the processor 3210 and / or the switch 3230 through the host bus interface. The host bus interface may be implemented as one of the above examples of the interface 3254. In one embodiment, the NIC 3240 may be integrated with at least one of the processor 3210, the switch 3230, and the storage device 3250.

[0171] Organic Relationship-Interface Operation

[0172] In the storage servers 3200 to 3200m or the application servers 3100 to 3100n, the processor may send a command to the storage devices 3150 to 3150n and 3250 to 3250m or the memories 3120 to 3120n and 3220 to 3220m and program or read data. In this case, the data may be data in which an error is corrected by an ECC engine. The data may be data on which a data bus inversion (DBI) operation or a data masking (DM) operation is performed, and may include cyclic redundancy coding (CRC) information. The data may be data encrypted for security or privacy.

[0173] The storage devices 3150 to 3150n and 3250 to 3250m may send control signals and command / address signals to the NAND flash memory devices 3252 to 3252m in response to a read command received from the processor. Therefore, when data is read from the NAND flash memory devices 3252 to 3252m, a read enable (RE) signal may be input as a data output control signal, and thus, the data may be output to the DQ bus. The data strobe signal DQS may be generated using the RE signal. Depending on the rising edge or falling edge of the write enable (WE) signal, the command and address signals may be latched in the page buffer.

[0174] Product section - Basic operation of SSD

[0175] The controller 3251 may control all operations of the storage device 3250. In one embodiment, the controller 3251 may include an SRAM. The controller 3251 may write data to the NAND flash memory device 3252 in response to a write command, or read data from the NAND flash memory device 3252 in response to a read command. For example, a write command and / or a read command may be provided from a processor 3210 of the storage server 3200, a processor 3210m of another storage server 3200m, or processors 3110 and 3110n of application servers 3100 and 3100n. The DRAM 3253 may temporarily store (or buffer) data to be written to the NAND flash memory device 3252 or data to be read from the NAND flash memory device 3252. And, the DRAM 3253 may store metadata. Here, the metadata may be user data or data generated by the controller 3251 for managing the NAND flash memory device 3252. The storage device 3250 may include a secure element (SE) for security or privacy.

[0176] According to an exemplary embodiment of the present disclosure, a data center system (e.g., 3000) is provided, comprising: a plurality of application servers (3100 to 3100n); and a plurality of storage servers (e.g., 3200 to 3200m), wherein each storage server comprises a storage device, wherein the storage device is configured to execute the method for model training as described above.

[0177] As is conventional in the field of the present disclosure, embodiments are described and shown in the accompanying drawings according to functional blocks, units and / or modules. It will be appreciated by those skilled in the art that these blocks, units and / or modules are physically implemented by electronic (or optical) circuits (such as logic circuits, discrete components, microprocessors, hard-wired circuits, memory elements, wiring connections, etc.), which can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. In the case where blocks, units and / or modules are implemented by microprocessors or the like, they can be programmed using software (e.g., microcode) to perform the various functions discussed herein, and they can be optionally driven by firmware and / or software. Optionally, each block, unit and / or module can be implemented by dedicated hardware, or implemented as a combination of dedicated hardware that performs some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) that performs other functions.

[0178] According to an embodiment of the present disclosure, a computer program product storing instructions is provided, wherein when the instructions are executed by a processor, the method for model training as described above is implemented.

[0179] According to an embodiment of the present disclosure, an electronic device is provided, including: a processor and a memory, storing a computer program, wherein when the computer program is executed by the processor, the method for model training as described above is implemented.

[0180] According to an embodiment of the present disclosure, a computer-readable storage medium storing a computer program may also be provided. When the program is run, the method for model training according to an embodiment of the present disclosure may be executed. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, for example, multimedia card, secure digital (SD) card and / or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and / or any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-temporary manner and provide the computer programs and any associated data, data files and / or data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In one example, the computer program and any associated data, data files and / or data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0181] According to the method and system for model training of the exemplary embodiments of the present disclosure, most of the parameters in the training of large deep learning models are offloaded, thereby reducing GPU memory consumption, and MS SSD is introduced instead of host RAM to offload parameters to further reduce costs; in addition, a layer-based grouping offloading strategy is proposed for parallel prefetching / reading / writing / weight updating based on multiple MS SSDs to avoid IO bottlenecks and training delays; in addition, the calculation of near data (weight update) is offloaded to MS SSD to avoid data transmission bottlenecks, and multi-scale parallel weight update is used to accelerate near data calculation and keep up with GPU training speed; in addition, backup-based weight transfer is proposed to avoid waiting for weight update before the next FP, and weight backup is also used for training crash recovery.

[0182] While the present disclosure has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims.

Claims

1. A training method for a deep learning model, comprising: The parameters of each layer of the deep learning model are grouped by a graphics processing unit (GPU); The GPU unloads the grouped parameters to a plurality of storage devices respectively, wherein each storage device stores a set of parameters for each layer; The deep learning model is trained by the GPU and the multiple storage devices in parallel.

2. The method according to claim 1, characterized in that The GPU and the plurality of storage devices perform training of the deep learning model in parallel, including: When performing forward propagation of the i-th layer of the deep learning model, the GPU loads the weights of the i+1-th layer of the deep learning model from the dynamic random access memory DRAM of the multiple storage devices into the memory of the GPU, where i is a positive integer; and The multiple storage devices pre-fetch the weights of the i+2th layer of the deep learning model from the NAND of the multiple storage devices to the DRAM.

3. The method according to claim 1, characterized in that The GPU and the plurality of storage devices perform training of the deep learning model in parallel, including: When performing the back propagation of the deep learning model, the GPU groups the gradients of each layer and writes the grouped gradients into the NANDs of the multiple storage devices respectively.

4. The method according to claim 1, characterized in that The GPU and the plurality of storage devices perform training of the deep learning model in parallel, including: The field programmable gate arrays (FPGAs) in the multiple storage devices execute weight updates of the deep learning model in parallel.

5. The method according to claim 4, characterized in that The field programmable gate arrays FPGA in the plurality of storage devices execute weight updates of the deep learning model in parallel, including: In the FPGA in each storage device, the weight update of the deep learning model is performed in a data parallel manner and a pipeline parallel manner.

6. The method according to claim 4, characterized in that Also includes: When executing the weight update of the deep learning model, the multiple storage devices respectively store the currently updated weights in another device; When the weight update is completed, the multiple storage devices delete the weights previously stored in the multiple storage devices and store the weights in the other device into the multiple storage devices respectively.

7. The method according to any one of claims 1 to 6, characterized in that The plurality of storage devices include a memory-semantic solid state drive MS SSD.

8. A deep learning model training system, comprising: Controller; Graphics Processing Unit GPU; as well as Multiple storage devices; The controller is configured as follows: Controlling the GPU to group parameters of each layer of the deep learning model; Controlling the GPU to unload the grouped parameters to the multiple storage devices respectively, wherein each storage device stores a set of parameters of each layer; Control the GPU and the multiple storage devices to perform training of the deep learning model in parallel.

9. The system according to claim 8, characterized in that The controller is configured as: When performing forward propagation of the i-th layer of the deep learning model, control the GPU to load the weight of the i+1-th layer of the deep learning model from the dynamic random access memory DRAM of the multiple storage devices into the memory of the GPU, where i is a positive integer; as well as Control the multiple storage devices to pre-fetch the weights of the i+2th layer of the deep learning model from the NAND of the multiple storage devices to the DRAM.

10. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images