A ZNS-based AI large model storage acceleration algorithm

Abstract

The present invention provides a ZNS-based AI large model storage acceleration algorithm. By defining the storage mode and state machine scheduling algorithm of the ZNS-based neural network learning algorithm, the ZNS-based SSD separately stores the parameters of each layer, reducing the device's demand for large-capacity memory storage, while improving the parameter import performance during the neural network algorithm learning process, reducing overall power consumption and whole-disk write amplification. The present invention combines the processing definition of the neural network learning algorithm with the host computer mode coordinated with the ZNS SSD, cooperates with the host computer to improve the computing performance of the model, and facilitates the upper-layer ecological software to mobilize the data of each Zone; the ZNS SSD cooperates with the host computer to define the timing of loading parameters for each layer, reducing the capacity demand for DRAM during the large model processing process. At the same time, the ZNS SSD enters a low-power mode when the model parameters do not need to be loaded to save the power consumption of the entire machine, which is convenient for promoting the application fields and scenarios of large models.

Description

Technical Field

[0001] The present invention belongs to the field of computer storage technology, and specifically relates to a ZNS-based AI large model storage acceleration algorithm. Background Art

[0002] AI applications are becoming increasingly widespread, and these applications rely on high-speed computing and storage of large models. Current mainstream storage systems often experience performance bottlenecks when handling large amounts of random data read and write. This can severely impact the responsiveness of AI applications and even lead to erroneous calculations, impacting the accuracy of large models.

[0003] Currently, large-model computations in the industry rely primarily on GPUs equipped with a high-capacity and high-speed NVIDA HBM chip as cache. While using high-specification HBM chips can temporarily address storage speed issues, the increasing number of parameters required for large models will drastically increase the memory capacity required. Furthermore, as HBM chip capacity increases, yield issues will arise, leading to increased production capacity and higher computational costs for large models.

[0004] Currently, mainstream large-scale models are primarily based on deep learning algorithms using deep neural networks (DNNs). DNNs are neural networks built on multiple hidden layers, each of which acts as both a transmitter and processor of information. They transform input data into more expressive feature representations through nonlinear activation functions. These continuous nonlinear transformations enable DNNs to capture the deep, complex characteristics of the input data.

[0005] Figure 1 The DNN deep learning algorithm shown above has multiple hidden layers with different parameters. The amount of storage space required by each hidden layer is related to the number of neurons in that hidden layer. This occupied storage space is often on the order of GB. Loading all layers into memory would increase the training cost of the large model. However, non-volatile storage (SSDs) are relatively inexpensive and can be considered for storage. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an AI large model storage acceleration algorithm based on ZNS, which is used to improve the reading and writing speed and storage performance of large model data.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a ZNS-based AI large model storage acceleration algorithm, comprising the following steps:

[0008] S1: Before training the large model, store the training parameters of the large model in layers in SSD;

[0009] S2: When training a large model, the host computer sets the timing of loading parameters for each layer and the delay between calling parameters for each layer through a sequence of instructions;

[0010] S3: The SSD activates the corresponding Zone state machine according to the received instruction sequence, and cooperates with the host computer to perform operations on the large model, including loading the training parameters of each layer of the model, writing the training parameters of each layer of the model, and formatting the data of that layer.

[0011] According to the above scheme, in step S1, the specific steps of hierarchical storage of parameters are as follows: defining the zone corresponding to each layer of parameters in the SSD, and at the same time, specifying the zone ID corresponding to the corresponding parameters in the host computer, storing the parameters of each layer separately, so that the data in the zone is written sequentially and there is no cross-storage between the parameters of each layer.

[0012] According to the above solution, in step S3, the specific steps of loading the model are:

[0013] At time T, the SSD schedules the state machine of the zone where the model parameters to be loaded in the specified layer set by the host computer software enters the Read state;

[0014] Load the model parameters from the SSD into the memory for training, and confirm whether the parameters need to be modified based on the training result data; if the parameters need to be modified, the original parameters will be invalidated and all parameters will be updated to the corresponding zone; if no modification is required, no update will be performed.

[0015] Furthermore, in step S3, after the model is loaded, the SSD enters a low-power Empty state.

[0016] According to the above solution, in step S3, the specific steps of writing the model are:

[0017] After model training is completed, the state machine of the SSD scheduling zone enters the Write state;

[0018] Write the model parameters to SSD.

[0019] Furthermore, in step S3, after the model data is written, the state machine of the Zone enters the Close mode to further reduce power consumption.

[0020] According to the above solution, in step S3, the specific steps of formatting are:

[0021] If all the data in a zone of the SSD is invalid, the zone's state machine enters the Format state;

[0022] After the Format is completed, it re-enters the Write state and performs the write model operation on the SSD.

[0023] According to the above solution, in step S3, after the calculation of the parameters of each layer of SSD is completed, the parameters of the current layer in the memory are cleared and the parameters of the next layer are imported.

[0024] A ZNS-based AI large model storage acceleration system includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of a ZNS-based AI large model storage acceleration algorithm.

[0025] A computer memory stores a computer program that can be executed by a computer processor, and the computer program executes a ZNS-based AI large model storage acceleration algorithm.

[0026] The beneficial effects of the present invention are:

[0027] 1. The present invention discloses a ZNS-based AI large model storage acceleration algorithm. By defining a storage mode and a state machine scheduling algorithm based on a ZNS-based neural network learning algorithm, the ZNS-based SSD separately stores the parameters of each layer, allowing data in the area to be written sequentially. This reduces write amplification problems on the device side, memory usage and latency, increases bandwidth, and improves write speeds. This improves the read and write speeds and storage performance of large model data.

[0028] 2. The present invention combines the processing definition of the neural network learning algorithm with the host computer mode of ZNS SSD. During training, the zone corresponding to each layer of parameters is defined, and the state machine of the corresponding zone is driven during training. This storage method cooperates with the host computer in the actual operation of large models to improve the computing performance of the model and facilitate the upper-level ecological software to mobilize the data of each zone.

[0029] 3. The ZNS SSD of the present invention cooperates with the host computer to define the timing of loading parameters for each layer, reducing the capacity demand for DRAM during large model processing and the hardware cost of training large models; at the same time, the ZNS SSD enters low-power mode when loading is not required to save power consumption of the entire machine, which facilitates the promotion of application fields and scenarios of large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the existing DNN deep learning algorithm diagram.

[0031] Figure 2 This is the existing SSD storage map.

[0032] Figure 3 This is a ZNS-based SSD storage diagram according to an embodiment of the present invention.

[0033] Figure 4 This is a state machine driven algorithm diagram based on the DNN deep learning algorithm in an embodiment of the present invention.

[0034] Figure 5 This is a flowchart of hierarchical storage and loading according to an embodiment of the present invention.

[0035] Figure 6 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] Example 1:

[0038] See also Figure 6 , the processing flow of the embodiment of the present invention is as follows:

[0039] ZNS (Zoned Name Space) is a new feature in the NVMe 2.0 protocol. Its greatest advantage is that it allows data in the zone to be written sequentially, reducing write amplification problems on the device side, memory usage and latency, and increasing bandwidth. At the same time, each zone can better cooperate with upper-layer ecological software, making it easier for upper-layer ecological software to mobilize data in each zone.

[0040] In the DNN deep learning algorithm, the parameters of each hidden layer need to be trained. Usually, the input is given, compared with the output after passing through the hidden layer, and then the parameters are adjusted and written to the non-volatile storage SSD.

[0041] Combining the characteristics of ZNS with DNN deep learning algorithms creates a ZNS-based SSD, storing parameters for each layer separately rather than intermixing them. Due to the characteristics of SSDs, if intermixing occurs, even if only one layer's parameters are changed during training, the parameters of other layers will need to be moved and rewritten to new addresses. However, a ZNS-based SSD can define a separate zone for each parameter, reducing write amplification during training at each layer while increasing write speeds.

[0042] from Figure 2 As can be seen from the figure, in traditional SSD, the parameters of all layers are mixed storage; Figure 3 In ZNSSSD, parameters for each layer are stored independently based on the zone division. This allows for modifying parameters for each layer without moving data from other layers, improving write performance during parameter training and reducing write latency.

[0043] During training, define the corresponding zone of each layer parameter. During training, it involves driving the state machine corresponding to the zone. The state machine driving algorithm based on the DNN deep learning algorithm is as follows: Figure 4 As shown in the figure, the DNN deep learning algorithm primarily loads model data from the SSD into memory, trains the model, and modifies some parameters. During this process, the SSD can be in the Empty state to save power. After model training is complete, the model parameters need to be written to the SSD. At this point, the SSD enters Write mode. If the data in a certain zone is completely invalid, it enters Format mode. After Format is complete, it re-enters Write mode and executes the write operation. Once the entire zone is written, the zone enters Close mode to achieve lower power consumption.

[0044] In addition to improving write performance and reducing write latency during training, this storage method can also improve the performance of model operations in actual large-scale model operations when combined with the host computer.

[0045] The computational processing flow for large models is that after each layer of calculation is completed, the parameters for the next layer are imported, and the parameters for the current layer are directly eliminated from memory. Therefore, the ZNS SSD works with the host computer to define the timing for loading parameters for each layer. The ZNS SSD significantly reduces the demand for DRAM capacity while having little impact on calculations. At the same time, the ZNS SSD enters low-power mode when loading is not required, saving power for the entire device.

[0046] Example 2:

[0047] A computer memory stores a computer program that can be executed by a computer processor, and the computer program executes a ZNS-based AI large model storage acceleration algorithm. Figure 5 The training parameters of the large model are stored in layers in the storage structure. For example, the parameters of Layer 1 are stored in Zone 1, the parameters of Layer 2 are stored in Zone 2, and so on. When training a large model, the delay between calling the parameters of each layer is planned in advance on the host computer and sent to the SSD through a custom instruction sequence. The SSD activates the corresponding Zone state machine according to the received instruction sequence and cooperates with the application layer of the host computer. For example, at time T1, when importing the parameters of Layer 1, the SSD enters the activated Read state of Zone 1, and enters low power consumption after the reading is completed; at time T2, the application needs to import the parameters of Layer 2. At this time, the SSD internally schedules the Zone 2 state machine to enter the activated Read state, and enters the low power consumption state, that is, the Empty state, after the parameter reading is completed, and so on.

[0048] Example 3:

[0049] A ZNS-based AI storage acceleration system includes a processor and a memory, wherein the memory stores computer instructions. The processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of a ZNS-based AI large model storage acceleration algorithm.

[0050] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A ZNS-based AI large model storage acceleration algorithm, characterized by: The following steps are involved: S1: Before training the large model, store the training parameters of the large model in layers in SSD; Define the zone corresponding to each layer of parameters in the SSD. At the same time, specify the zone ID corresponding to the corresponding parameters in the host computer. Store the parameters of each layer separately so that the data in the zone is written sequentially and there is no cross-storage between the parameters of each layer. S2: When training a large model, the host computer sets the timing of loading parameters for each layer and the delay between calling parameters for each layer through a sequence of instructions; S3: The SSD activates the corresponding Zone state machine according to the received instruction sequence, and cooperates with the host computer to perform operations on the large model, including loading the training parameters of each layer of the model, writing the training parameters of each layer of the model, and formatting the data of that layer.

2. The ZNS-based AI large model storage acceleration algorithm according to claim 1, characterized in that: In step S3, the specific steps of loading the model are: At time T, the state machine of the zone where the model parameters of the specified layer set by the host computer software are located enters the Read state; Load the model parameters from the SSD into the memory for training, and confirm whether the parameters need to be modified based on the training results. If the parameters need to be modified, the original parameters will be invalidated and all parameters will be updated to the corresponding zone. If no modification is required, no update is performed.

3. The ZNS-based AI large model storage acceleration algorithm according to claim 2, characterized in that: In step S3, after the model is loaded, the SSD enters the low-power Empty state.

4. The ZNS-based AI large model storage acceleration algorithm according to claim 1, characterized in that: In step S3, the specific steps of writing the model are: After model training is completed, the state machine of the SSD scheduling zone enters the Write state; Write the model parameters to SSD.

5. The ZNS-based AI large model storage acceleration algorithm according to claim 4 is characterized by: In step S3, when the model data is written, the state machine of the Zone enters the Close mode to further reduce power consumption.

6. The ZNS-based AI large model storage acceleration algorithm according to claim 1, characterized in that: In step S3, the specific steps of formatting are: If all the data in a zone of the SSD is invalid, the zone's state machine enters the Format state; After the Format is completed, it re-enters the Write state and performs the write model operation on the SSD.

7. The ZNS-based AI large model storage acceleration algorithm according to claim 1, characterized in that: In step S3, after the calculation of the parameters of each layer of the SSD is completed, the parameters of the current layer in the memory are cleared and the parameters of the next layer are imported.

8. A ZNS-based AI large model storage acceleration system, comprising a processor and memory, characterized by: The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of a ZNS-based AI large model storage acceleration algorithm as described in any one of claims 1 to 7.

9. A computer memory, characterized in that: A computer program executable by a computer processor is stored therein, and the computer program executes a ZNS-based AI large model storage acceleration algorithm as described in any one of claims 1 to 7.

Images