AI large model storage acceleration algorithm based on ZNS

Abstract

According to the ZNS-based AI large model storage acceleration algorithm provided by the invention, the storage mode of the ZNS-based neural network learning algorithm and the state machine scheduling algorithm are defined, and the ZNS-based SSD independently stores parameters of each layer, so that the requirement of equipment for large-capacity memory storage is reduced, and the storage speed of the equipment is improved. Meanwhile, the parameter import performance in the neural network algorithm learning process is improved, and the overall power consumption and the whole disk write amplification are reduced. According to the method, an upper computer mode matched with ZNS SSD is defined in combination with the processing process of a neural network learning algorithm, the operation performance of a model is improved in cooperation with an upper computer, and upper-layer ecological software can conveniently mobilize data of each Zone; the ZNS SSD is matched with the upper computer to define the time sequence of the loading parameters of each layer, the capacity requirement for the DRAM in the large model processing process is reduced, meanwhile, the ZNS SSD enters the low-power-consumption mode when the model parameters do not need to be loaded so as to save the power consumption of the whole machine, and the application field and scene of the large model can be conveniently popularized.

Description

Technical Field

[0001] The present invention belongs to the field of computer storage technology, and in particular relates to a ZNS-based AI large model storage acceleration algorithm. Background Art

[0002] AI applications are becoming more and more widespread, and these applications all rely on high-speed large-model computing and storage. Current mainstream storage will have performance bottlenecks when facing large amounts of random data reading and writing, which will seriously affect the response of AI applications and even calculate incorrect results, affecting the accuracy of large models.

[0003] At present, the calculation of large models in the industry basically relies on NVIDA's built-in high-capacity and high-speed HBM chip as a GPU cache. Although the use of high-specification HBM chips can temporarily solve the problem of storage speed, as the parameters required for large models increase in the future, the memory capacity required for large models will also increase sharply; and as the capacity of HBM chips increases, the yield of HBM chips will lead to production capacity problems, and also cause the cost of large model calculations to increase.

[0004] The current mainstream large models are mainly based on deep learning algorithms based on deep neural networks (DNNs). Deep neural networks are neural networks built on multiple hidden layers, and each layer plays the role of information transmitter and processor. The input data is converted into more expressive feature representations through nonlinear activation functions. These continuous nonlinear transformations enable DNNs to capture the deep and complex features of the input data.

[0005] Figure 1 In the DNN deep learning algorithm shown in the figure, there are multiple hidden layers in the middle, and different hidden layer parameters are different. The size of the storage space required by the hidden layer is related to the number of neurons in the current hidden layer. This occupied storage space is often in the order of GB. If all layers are loaded into the memory space, the training cost of the large model will increase; however, the storage cost of non-volatile storage SSD is low, so SSD can be considered for storage. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an AI large model storage acceleration algorithm based on ZNS, which is used to improve the reading and writing speed and storage performance of large model data.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a ZNS-based AI large model storage acceleration algorithm, comprising the following steps:

[0008] S1: Before training the large model, store the training parameters of the large model in layers in SSD;

[0009] S2: When training a large model, the host computer sets the timing of loading parameters for each layer and the delay between calling parameters for each layer through instruction sequences;

[0010] S3: The SSD activates the corresponding Zone state machine according to the received instruction sequence, and cooperates with the host computer to perform operations on the large model, including loading the training parameters of each layer of the model, writing the training parameters of each layer of the model, and formatting the data of this layer.

[0011] According to the above scheme, in step S1, the specific steps of hierarchical storage parameters are: defining the zone corresponding to each layer of parameters in the SSD, and at the same time specifying the zone ID corresponding to the corresponding parameters in the host computer, storing the parameters of each layer separately, so that the data in the area is written sequentially, and no cross storage is generated between the parameters of each layer.

[0012] According to the above scheme, in step S3, the specific steps of loading the model are:

[0013] At time T, the state machine of the zone where the model parameters to be loaded at the specified layer set by the host computer software are located in the SSD scheduling enters the Read state;

[0014] Load the model parameters from the SSD into the memory for training, and confirm whether the parameters need to be modified based on the training result data; if the parameters need to be modified, the original parameters will be invalidated and all parameters will be updated to the corresponding Zone; if no modification is required, no update will be performed.

[0015] Furthermore, in the step S3, after the model is loaded, the SSD enters a low-power Empty state.

[0016] According to the above scheme, in step S3, the specific steps of writing the model are:

[0017] After the model training is completed, the state machine of the SSD scheduling zone enters the Write state;

[0018] Write the model parameters to SSD.

[0019] Furthermore, in step S3, when the model data is written, the state machine of the Zone enters the Close mode to further reduce power consumption.

[0020] According to the above scheme, in step S3, the specific steps of formatting are:

[0021] If all the data in a zone of the SSD is invalid, the state machine of the zone enters the Format state;

[0022] After the Format is completed, it re-enters the Write state and performs operations on the SSD to write the model.

[0023] According to the above scheme, in step S3, after the calculation of the parameters of each layer of the SSD is completed, the parameters of the current layer in the memory are cleared and the parameters of the next layer are imported.

[0024] A ZNS-based AI large model storage acceleration system, the system comprises a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps of a ZNS-based AI large model storage acceleration algorithm.

[0025] A computer memory stores a computer program that can be executed by a computer processor, and the computer program executes a ZNS-based AI large model storage acceleration algorithm.

[0026] The beneficial effects of the present invention are:

[0027] 1. The present invention discloses a ZNS-based AI large model storage acceleration algorithm, which defines a storage mode of a ZNS-based neural network learning algorithm and a state machine scheduling algorithm. The ZNS-based SSD stores the parameters of each layer separately, so that the data in the area is written sequentially, reducing the write amplification problem on the device side, the use of memory and latency, increasing bandwidth, and improving the writing speed; thus, the function of improving the reading and writing speed and storage performance of large model data is realized.

[0028] 2. The present invention combines the processing definition of the neural network learning algorithm with the host computer mode that cooperates with ZNS SSD. During training, the Zone corresponding to each layer of parameters is defined, and the state machine of the corresponding Zone is driven during training. This storage method cooperates with the host computer in the actual operation of the large model to improve the computing performance of the model, and facilitates the upper-level ecological software to mobilize the data of each Zone.

[0029] 3. The ZNS SSD of the present invention cooperates with the host computer to define the timing of loading parameters for each layer, which reduces the capacity demand for DRAM during the processing of large models and reduces the hardware cost of training large models; at the same time, the ZNS SSD enters a low-power mode when loading is not required to save the power consumption of the entire machine, which is convenient for promoting the application fields and scenarios of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the existing DNN deep learning algorithm diagram.

[0031] Figure 2 This is the existing SSD storage map.

[0032] Figure 3 This is a ZNS-based SSD storage diagram of an embodiment of the present invention.

[0033] Figure 4 It is a state machine driven algorithm diagram based on the DNN deep learning algorithm of an embodiment of the present invention.

[0034] Figure 5 The figure is a flowchart of hierarchical storage and loading according to an embodiment of the present invention.

[0035] Figure 6 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] Embodiment 1:

[0038] See also Figure 6 , the processing flow of the embodiment of the present invention is as follows:

[0039] ZNS (Zoned Name Space) is a new feature in the NVMe2.0 protocol. Its biggest advantage is that it keeps the data in the zone written sequentially, reducing the write amplification problem on the device side, memory usage and latency, and increasing bandwidth. At the same time, each Zone cooperates better with the upper-layer ecological software, making it easier for the upper-layer ecological software to mobilize the data of each Zone.

[0040] In the DNN deep learning algorithm, the parameters of each hidden layer need to be trained, usually by giving an input, comparing the output after passing through the hidden layer, and then adjusting the parameters and writing the adjusted parameters to the non-volatile storage SSD.

[0041] Combining the characteristics of ZNS with the DNN deep learning algorithm, a ZNS-based SSD is formed, which stores the parameters of each layer separately instead of mixed storage. Due to the characteristics of SSD, if mixed storage occurs, even if only one layer of parameter training is changed, the parameters of other layers will need to be moved and rewritten to a new address; if it is a ZNS-based SSD, a zone can be defined for each parameter separately, so that write amplification can be reduced when training each layer, while the writing speed can be improved.

[0042] from Figure 2 As can be seen from the figure, in traditional SSDs, the parameters of all layers are stored in a mixed manner; Figure 3 In ZNSSSD, the parameters of each layer are stored independently according to the different divisions of the zones. This way, when modifying the parameters of each layer, there is no need to move the data of other layers, which improves the writing performance during parameter training and reduces the writing latency.

[0043] During training, define the corresponding zone of each layer parameter. During training, the state machine driving the corresponding zone is involved. The state machine driving algorithm based on the DNN deep learning algorithm is as follows: Figure 4 As shown. The DNN deep learning algorithm mainly loads the model data from the SSD to the memory, trains the model, and modifies some parameters. During this process, the SSD can be in the Empty state to save power consumption. After the model training is completed, the model parameters need to be written to the SSD. At this time, the SSD needs to enter the Write mode. If a part of the Zone space data is completely invalid, it enters the Format mode. After the Format is completed, it re-enters the Write mode and executes the write action. After the entire part of the data is written, the Zone space enters the Close mode to enter a lower power consumption mode.

[0044] In addition to improving write performance and reducing write latency during training, this storage method can also improve the performance of model operations in conjunction with the host computer in actual large model operations.

[0045] The operation process of the large model is that after each layer of calculation is completed, the parameters of the next layer are imported, and the parameters of the current layer are directly eliminated from the memory. Therefore, ZNS SSD cooperates with the host computer to define the timing of loading parameters for each layer. ZNS SSD significantly reduces the demand for DRAM capacity with little impact on calculation; at the same time, ZNS SSD enters low power mode when loading is not required, saving power consumption of the entire machine.

[0046] Embodiment 2:

[0047] A computer memory stores a computer program executable by a computer processor, wherein the computer program executes a ZNS-based AI large model storage acceleration algorithm. Figure 5 The training parameters of the large model are stored in layers in the storage structure. For example, the parameters of Layer 1 are stored in Zone 1, the parameters of Layer 2 are stored in Zone 2, and so on. When training a large model, the delay between the calls of each layer of parameters is planned in advance on the host computer and sent to the SSD through a custom instruction sequence. The SSD activates the corresponding Zone state machine according to the received instruction sequence and cooperates with the host computer application layer. For example, at time T1, when the parameters of Layer 1 are imported, the SSD enters the activated Read state of Zone 1, and enters low power consumption after the reading is completed; at time T2, the application needs to import the parameters of Layer 2. At this time, the SSD internally schedules the Zone 2 state machine to enter the activated Read state, and enters the low power consumption state, that is, the Empty state, after the parameter reading is completed, and so on.

[0048] Embodiment 3:

[0049] A ZNS-based AI storage acceleration system includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of a ZNS-based AI large model storage acceleration algorithm.

[0050] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A ZNS-based AI large model storage acceleration algorithm, characterized by: The following steps are involved: S1: Before training the large model, store the training parameters of the large model in layers in SSD; S2: When training a large model, the host computer sets the timing of loading parameters for each layer and the delay between calling parameters for each layer through instruction sequences; S3: The SSD activates the corresponding Zone state machine according to the received instruction sequence, and cooperates with the host computer to perform operations on the large model, including loading the training parameters of each layer of the model, writing the training parameters of each layer of the model, and formatting the data of this layer.

2. According to the ZNS-based AI large model storage acceleration algorithm of claim 1, it is characterized by: In step S1, the specific steps of storing parameters in layers are as follows: defining the zone corresponding to each layer of parameters in the SSD, and at the same time specifying the zone ID corresponding to the corresponding parameters in the host computer, storing the parameters of each layer separately, so that the data in the area is written sequentially, and no cross storage is generated between the parameters of each layer.

3. According to the ZNS-based AI large model storage acceleration algorithm of claim 1, it is characterized by: In step S3, the specific steps of loading the model are: At time T, the state machine of the zone where the model parameters of the specified layer set by the upper computer software are located in the SSD scheduling enters the Read state; Load the model parameters from the SSD into the memory for training, and confirm whether the parameters need to be modified based on the training result data; if the parameters need to be modified, the original parameters will be invalidated and all parameters will be updated to the corresponding Zone; If no modification is required, then no update is performed.

4. According to the ZNS-based AI large model storage acceleration algorithm of claim 3, it is characterized by: In the step S3, after the model is loaded, the SSD enters the low-power Empty state.

5. According to the ZNS-based AI large model storage acceleration algorithm of claim 1, it is characterized by: In step S3, the specific steps of writing the model are: After the model training is completed, the state machine of the SSD scheduling zone enters the Write state; Write the model parameters to SSD.

6. According to the ZNS-based AI large model storage acceleration algorithm of claim 5, it is characterized by: In the step S3, when the model data is written, the state machine of the Zone enters the Close mode to further reduce power consumption.

7. The AI ​​large model storage acceleration algorithm based on ZNS according to claim 1 is characterized in that: In step S3, the specific steps of formatting are: If all the data in a zone of the SSD is invalid, the state machine of the zone enters the Format state; after the Format is completed, it re-enters the Write state to perform the write model operation on the SSD.

8. The AI ​​large model storage acceleration algorithm based on ZNS according to claim 1 is characterized in that: In the step S3, after the calculation of the parameters of each layer of the SSD is completed, the parameters of the current layer in the memory are cleared and the parameters of the next layer are imported.

9. A ZNS-based AI large model storage acceleration system, the system comprising a processor and a memory, characterized in that: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of a ZNS-based AI large model storage acceleration algorithm as described in any one of claims 1 to 8.

10. A computer memory, characterized in that: A computer program executable by a computer processor is stored therein, and the computer program executes a ZNS-based AI large model storage acceleration algorithm as described in any one of claims 1 to 8.

Images