Single-GPU large model training method and system based on multiple SSDs

By equipping with multiple NVMe SSDs and asynchronous communication strategies in single GPU systems, the problems of storage bandwidth differences and communication complexity in training single GPU large model are solved, and the ability to efficiently train large language models on consumer GPUs is realized, improving the ability to efficiently train large language models on consumer GPUs, Training performance and model size are achieved while reducing costs.

CN118939434AActive Publication Date: 2024-11-12ZHEJIANG UNIV

Patent Information

Application Number
CN202411412146.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-11
Publication Date
2024-11-12
Estimated Expiration
2044-10-11

AI Technical Summary

Technical Problem

In the prior art, single GPU big model training has caused a difference in CPU-GPU and SSD-CPU bandwidth due to storage limitations, resulting in a decline in training performance. The GPU Direct Storage technology of consumer GPUs is unavailable, which increases communication complexity and is difficult to budget. Efficient training of large language models in limited environments.

Method used

By carrying multiple NVMe SSDs, the SSD-CPU bandwidth is improved, asynchronous communication and buffer management strategies are adopted, and the system information is recorded in combination with PyTorch's Hook mechanism, and the buffer size is dynamically adjusted to realize parallel execution of computing and communication, hiding communication overhead, and Active offloading technology is introduced to reduce gradient redundant transmission.

Benefits of technology

It realizes efficient training of large-scale language models on a single consumer GPU, significantly improving training performance and trainable model scale, reducing training costs, and avoiding redundant transmission problems caused by passive gradient unloading.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939434A_ABST
    Figure CN118939434A_ABST
Patent Text Reader

Abstract

The invention discloses a single GPU large model training method and system based on multiple SSDs, and belongs to the technical field of model heterogeneous training. Through software and hardware collaborative design carrying multiple SSDs and a single GPU, a space is additionally applied on the CPU and the GPU for asynchronous communication, overhead brought by communication is hidden while calculation is not affected, the problem that the training performance is reduced due to the bandwidth difference between the SSD-CPU and the CPU-GPU is solved, the trainable model scale and the training efficiency can be greatly improved, and the training efficiency is improved. Meanwhile, the training cost is reduced. Besides, an active unloading technology is introduced in a large model heterogeneous training process, reverse calculation on the GPU and optimizer calculation on the CPU are parallel by using a multi-process technology, and meanwhile, the optimizer calculation on the CPU and communication between SSD-CPU are parallel, so that the overhead caused by communication and calculation serialization is reduced, and the gradient redundancy transmission problem is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of model heterogeneous training, and specifically relates to a single GPU large model training method and system based on multiple SSDs. Background Art

[0002] Large language models (LLMs) such as ChatGPT have brought great value to the world, and current academic researchers have a strong demand for fine-tuning LLMs. The largest open-source LLM has grown to 405B, far exceeding the device memory of the GPU. Ordinary researchers usually aggregate CPU memory and SSD storage for heterogeneous training because the cost of aggregating multiple GPUs is high.

[0003] The data in large model training mainly involves model state (parameters, gradients, optimizer state) and activation values. For a 70B model, the model state occupies 1.5TB, and the activation value is linearly related to the number of tokens in each iteration of training. Even 1.5TB of storage far exceeds the total memory capacity of a single GPU memory and an ordinary server. Therefore, for researchers with limited budgets, it is necessary to introduce cheap SSD storage.

[0004] However, simply introducing SSD storage will bring significant performance issues. The bandwidth between CPU and GPU (PCIe 4.032GB / s) is significantly lower than the memory bandwidth (RTX4090, 1TB / s), so the training performance of CPU-GPU two-level storage will be constrained by the slow communication between CPU and GPU. The SSD-CPU bandwidth (P5510, 6GB / s) is even lower, further increasing the communication pressure. At the same time, because the GPU Direct Storage technology of consumer-grade GPUs is not available, an additional path is added, making the communication link more complicated. Summary of the invention

[0005] The purpose of the present invention is to solve the problems existing in the prior art and to propose a single GPU large model training method and system based on multiple SSDs. The present invention increases the bandwidth of the SSD-CPU and expands the storage space by carrying multiple NVMe SSDs, so that a large language model can be trained on a single consumer-grade GPU of a commodity server and the training performance can be improved as much as possible.

[0006] The specific technical solutions adopted by the present invention are as follows:

[0007] In a first aspect, the present invention provides a single GPU large model training method based on multiple SSDs, wherein the training method is implemented based on a server equipped with a single GPU and multiple SSDs; the training method comprises the following steps:

[0008] S1. On the GPU, the model parameters of the large model to be trained are initialized in FP16 format, and the initialized model parameters are transferred to the SSD for storage; then the model parameters in FP16 format are read from the SSD to the CPU of the server, and the corresponding model parameter copies in FP32 format are initialized on the CPU, and the gradients and optimizer states are initialized based on the model parameter copies in FP32 format, and then the model parameters and optimizer states in FP32 format initialized on the CPU are transferred to the SSD for storage;

[0009] S2. Perform the first training of the training iteration process on the initialized large model to be trained in a manner that all calculations and communications are performed serially, and record and analyze system information to determine the free memory of the GPU and CPU, the calculation ratio, the communication ratio, and the parameter amount of each of all network layers; then apply for a buffer on the CPU and GPU that does not exceed the free memory, and the buffer size is not less than the parameter amount of the network layer with the largest parameter amount in the large model to be trained;

[0010] S3, in a way that computation and communication are performed asynchronously, in a two-stage mode consisting of a forward propagation stage and a backpropagation-optimizer update parallel stage, the GPU performs the next training iteration of the training large model to be trained, while recording system information;

[0011] In the forward propagation stage, the GPU updates the model parameters of the large model to be trained layer by layer, and in the process of executing the forward calculation of the current network layer, the GPU needs to read the FP16 format model parameters of the subsequent network layers from the SSD in advance through the CPU until the buffer on the GPU is full. At the same time, the GPU needs to unload the calculated gradient checkpoints of the previous layer to the SSD through the CPU; after the forward calculation of each network layer is completed, the GPU needs to immediately release the model parameters of this network layer and start the forward calculation of the next network layer;

[0012] In the back propagation-optimizer update parallel phase, the GPU recalculates and back propagates the large model to be trained layer by layer, while the CPU updates the optimizer state layer by layer; when the GPU is performing the recalculation and back propagation of the current network layer, the GPU needs to read the FP16 format model parameters and gradient checkpoints of the subsequent network layers from the SSD in advance through the CPU until the buffer on the GPU is full, and at the same time, the GPU needs to unload the gradients obtained by the back propagation of the previous network layer to the CPU; after the recalculation and back propagation of each network layer are completed, the GPU needs to immediately release the model parameters and gradient checkpoints of this network layer, and start the recalculation and back propagation of the next network layer; when the CPU is performing the optimizer state update of the current network layer, it needs to write the optimizer state of the previous layer updated back to the SSD, and at the same time read the optimizer state of the next layer from the SSD in advance and generate a copy of the model parameters in FP32 format, so as to start the optimizer state update of the next layer immediately after receiving the gradient of the next layer unloaded by the GPU;

[0013] S4. Repeat S3 continuously until all training rounds of the training iteration process are completed; and before each repetition of S3, analyze whether it is necessary to expand the buffer size on the CPU and GPU according to the system information recorded in the previous training process, so as to maximize the training performance.

[0014] As a preferred embodiment of the first aspect, the GPU and SSD are connected to the PCIe slot of the server through a PCIe interface, and all SSDs receive data transmitted by the GPU and CPU according to the storage strategy of the server and perform persistent storage.

[0015] As a preferred embodiment of the above-mentioned first aspect, the server records system information in each round of training process through the Hook mechanism of PyTorch, including the free memory amount on the GPU and CPU respectively, the bandwidth of communication between the CPU and GPU, the computing time required for forward calculation of each network layer on the GPU, the computing time required for reverse calculation of each network layer on the GPU, and the parameter amount of each network layer in the large model to be trained.

[0016] As a preferred embodiment of the above-mentioned first aspect, before each repetition of S3, it is necessary to analyze whether the computing proportion of the server is greater than the communication proportion based on the system information recorded in the previous training process. If so, the buffer size on the CPU and GPU is expanded according to the preset increment before repeating S3; after the training performance of the server reaches its peak or any one of the CPU and GPU exceeds the free memory, the buffer will no longer be expanded.

[0017] As a preferred embodiment of the first aspect, the GPU is a consumer-grade GPU.

[0018] As a preferred embodiment of the above-mentioned first aspect, the GPU does not persistently store any data, but only saves temporary data required for forward and reverse calculations; the CPU does not persistently store any data, but only saves temporary data required for optimizer updates; the SSD persistently stores various types of data required for model training, including model parameters in FP16 format and optimizer status data.

[0019] In a second aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the single GPU large model training method based on multiple SSDs as described in any one of the first aspects above.

[0020] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the single GPU large model training method based on multiple SSDs as described in any one of the first aspects above is implemented.

[0021] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0022] The memory is used to store computer programs;

[0023] The processor is used to implement the single GPU large model training method based on multiple SSDs as described in any one of the first aspects above when executing the computer program.

[0024] In the fifth aspect, the present invention provides a single GPU large model training system based on multiple SSDs, which includes a server and a single GPU and multiple SSDs installed on the server; a large model training control program is running on the server, and the large model training program is used to implement the single GPU large model training method based on multiple SSDs as described in any one of the first aspects above.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention provides a single GPU large model training method and system based on multiple SSDs. Through the software and hardware collaborative design of multiple NVMe SSDs and a single consumer-grade GPU, an additional space is applied on the CPU and GPU for asynchronous communication. The overhead caused by communication is hidden without affecting the calculation, and the problem of bandwidth difference between SSD-CPU and CPU-GPU leading to decreased training performance is solved. The scale of trainable models and training efficiency can be greatly improved, while reducing training costs.

[0027] In addition, the present invention introduces active offloading technology in the process of large-model heterogeneous training, uses multi-process technology to parallelize the reverse calculation on the GPU and the optimizer calculation on the CPU, and parallelizes the optimizer calculation on the CPU and the communication between the SSD-CPU, thereby reducing the overhead caused by serial communication and calculation, and avoiding the problem of redundant gradient transmission caused by passive gradient offloading. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the steps of a single GPU large model training method based on multiple SSDs;

[0029] Figure 2 A schematic diagram of the data computation and communication mechanism in the existing large-model heterogeneous training method;

[0030] Figure 3 A schematic diagram of the data calculation and communication mechanism in the large model heterogeneous training method proposed in the present invention;

[0031] Figure 4 This is the training throughput test result in the embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0033] In the description of the present invention, it is to be understood that when an element is considered to be "connected" to another element, it may be directly connected to the other element or indirectly connected, that is, there are intermediate elements. On the contrary, when an element is said to be "directly" connected to another element, there are no intermediate elements.

[0034] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0035] In a preferred embodiment of the present invention, a single GPU large model training method based on multiple SSDs is provided, and the training method is implemented based on a server equipped with a single GPU and multiple SSDs, wherein the server has its own CPU. The CPU, GPU and SSD in the server constitute a heterogeneous training system for large models, which increases the bandwidth of the SSD-CPU and expands the storage space by carrying multiple SSDs, so that a large language model can be trained on a single consumer-grade GPU of a commercial server and the training performance can be improved as much as possible.

[0036] The GPU of the embodiment of the present invention can be implemented by a consumer-grade GPU. Even if the GPU DirectStorage technology of such a GPU is not available, it can still be used to efficiently train large models with a large number of parameters. Of course, the present invention is not limited to the use of consumer-grade GPUs, and the present invention is also applicable to other professional-grade GPUs and data center-grade GPUs. The SSD of the embodiment of the present invention can be an NVMe SSD.

[0037] It should be noted that when installing GPU and SSD on the server, the PCIe interface is the most common and recommended method because it provides a high-speed data transmission channel and wide compatibility. However, in other embodiments, depending on the specific configuration and requirements of the server, other interfaces such as SATA, SAS, M.2 or U.2 may also be considered to install the SSD. For the GPU, although the PCIe interface has almost become a standard configuration, in rare cases there may be other interface installation methods. In an embodiment of the present invention, considering the efficiency and compatibility of data transmission, the GPU and SSD are connected to the PCIe slot of the server through the PCIe interface, and all SSDs receive the data transmitted by the GPU and CPU according to the storage policy of the server and perform persistent storage.

[0038] The present invention is based on the above-mentioned heterogeneous training system, and mainly improves the data storage and calculation methods. In terms of data storage, due to the scale of LLM, the model parameters, gradients, activation values ​​and optimizer states generated during training far exceed the GPU video memory capacity and CPU memory capacity, so a heterogeneous storage method is adopted in the model training process: in the GPU, no data is persistently stored, only temporary data required for forward propagation and reverse propagation calculations are saved; in the CPU, no data is persistently stored, only temporary data required for optimizer updates are saved; in the SSD, FP16 format model parameters, activation values ​​and optimizer states and other types of data required for model training are persistently stored. In terms of calculation methods, in order to reduce the data transfer of optimizer states between different devices, a heterogeneous calculation method is also adopted, and forward and reverse calculations are completed on the GPU, and optimizer updates are completed on the CPU. Through Python's multi-process technology, reverse calculations on the GPU and optimizer calculations on the CPU are realized in parallel.

[0039] like Figure 1 As shown, the large model training method based on the above heterogeneous training system is described in detail below. The training method includes the following steps:

[0040] S1. On the GPU, the model parameters of the large model to be trained are initialized in FP16 format, and the initialized model parameters are transferred to the SSD for storage; then the model parameters in FP16 format are read from the SSD to the CPU of the server, and the corresponding model parameter copies in FP32 format are initialized on the CPU, and the gradients and optimizer states are initialized based on the model parameter copies in FP32 format, and then the model parameters and optimizer states in FP32 format initialized on the CPU are transferred to the SSD for storage.

[0041] The above step S1 corresponds to an initialization phase of the system, which is mainly to gradually initialize all training-related data on heterogeneous storage. Because a single GPU and CPU cannot complete the initialization alone due to storage limitations, they need to be processed with the help of SSD. In an embodiment of the present invention, the above step S1 of the initialization phase can be specifically implemented through the following sub-steps S11 to S13:

[0042] S11. Initialize model parameters: Since the large model parameters are too large to be initialized as a whole, the model parameters of the large model are initialized layer by layer using the FP16 format on the GPU, and are transferred layer by layer to the SSD through the CPU for management.

[0043] S12, initialize the optimizer state: In order to ensure the training accuracy of the large model, the CPU needs to use the model parameters in FP32 format to calculate the gradient. Therefore, after S11 is executed, the model parameters in FP16 format need to be read from SSD to CPU layer by layer, and the corresponding model parameter copies in FP32 format are initialized. The gradients are initialized based on the model parameters in FP32 format, and then the Adam operation is performed to initialize the optimizer state of the corresponding parameters. Finally, the initialized FP32 format model parameters and optimizer state are transferred to SSD for management.

[0044] S13. Initialize subprocess: During the large model training process, a process needs to be set up on the CPU to implement the optimizer update, so the optimizer update process on the CPU also needs to be initialized.

[0045] S2. Perform the first training iteration of the training process on the initialized large model to be trained in a manner that all calculations and communications are performed serially, and record and analyze system information to determine the free memory of the GPU and CPU, the calculation ratio, the communication ratio, and the parameter size of all network layers; then apply for a buffer on the CPU and GPU that does not exceed the free memory, and the buffer size is not less than the parameter size of the network layer with the largest parameter size in the large model to be trained.

[0046] The above step S2 corresponds to an analysis phase of the system. The analysis phase mainly collects and analyzes the calculation and communication data of the system during the training process, and then determines the total amount of data unloading and the communication and calculation parallel strategy based on these data. In an embodiment of the present invention, the above step S2 of the analysis phase can be specifically implemented through the following sub-steps S21 to S23:

[0047] S21: During the first training of the model training iteration, all calculations and communications are performed serially, including calculations on the GPU and CPU, so as to accurately obtain relevant data in the training process.

[0048] S22: The server uses PyTorch's Hook mechanism to record system information during the actual training process of S21, including the free memory on the GPU and CPU, the bandwidth of communication between the CPU and GPU, the computing time required for forward calculation of each network layer on the GPU, the computing time required for reverse calculation of each network layer on the GPU, and the number of parameters of each network layer in the large model to be trained. After recording the above system information, the system's computing ratio (the sum of the computing time of all network layers divided by the sum of the computing and communication time) and communication ratio (the sum of the communication time of all network layers divided by the sum of the computing and communication time) can be determined. The communication time of each network layer can be obtained by dividing the number of parameters of this network layer by the bandwidth of communication between the CPU and GPU.

[0049] S23. Apply for a buffer on each of the CPU and GPU. The size of the buffer on the CPU does not exceed the amount of free memory on the CPU, but is not less than the size of the parameters of the network layer with the largest number of parameters in the large model to be trained. Similarly, the size of the buffer on the GPU does not exceed the amount of free memory on the GPU, but is not less than the size of the parameters of the network layer with the largest number of parameters in the large model to be trained. Since the size of the buffer will be continuously adjusted in the future to achieve optimal performance, when the buffer is initially applied for on the CPU and GPU, the size can be equal to the size of the parameters of the network layer with the largest number of parameters in the large model to be trained.

[0050] It should also be noted that the lower limit of the buffer on the CPU and GPU cannot be lower than the parameter size of the network layer with the largest number of parameters in the large model to be trained. If the free memory on the CPU or GPU is lower than the parameter size of the network layer with the largest number of parameters in the large model to be trained, the training method of the present invention will not be applicable.

[0051] S3. In a two-stage mode consisting of a forward propagation stage and a backpropagation-optimizer update parallel stage, the GPU performs the next training iteration of the training large model in an asynchronous manner of computing and communication, while recording system information.

[0052] It should be noted that the asynchronous execution of calculation and communication in the present invention means that when the CPU or GPU in the server executes the relevant calculation of one network layer, the relevant data of other previous and subsequent network layers are transmitted through communication links, so that the calculation and communication processes are executed in parallel as much as possible, reducing the time loss caused by waiting for calculation or waiting for communication. In the entire model training iteration process, except for the first training process which uses the serial execution of calculation and communication, the other iteration rounds can use the asynchronous execution of calculation and communication.

[0053] Figure 2It is a traditional large-model heterogeneous training system, which is mainly divided into three stages: forward propagation stage, back propagation stage and optimizer update stage. The calculations in the three stages are completely serial. Under the traditional approach, when the optimizer is updated, the communication and calculation on the GPU side are completely idle; at the same time, the gradient checkpoint is only unloaded to the CPU, and it is designed around a single SSD. The heterogeneous training system proposed in this invention is Figure 3 As shown, it is mainly divided into the forward propagation stage and the back propagation-optimizer update parallel stage. Compared with the traditional approach, the optimizer update on the CPU and the back propagation on the GPU are executed in parallel. Therefore, the gradient checkpoint in the present invention can support further offloading to the SSD, and is designed around a RAID array composed of multiple SSDs; at the same time, the analysis and optimization process of the software and hardware information in the analysis stage can be combined to adjust the communication mode during the training process. Figure 2 and Figure 3 Medium 16 Represents the model parameters in FP16 format, A 16 Represents the gradient checkpoint in FP16 format, G 16 Represents the gradient in FP16 format, P 32 Represents the model parameters in FP32 format, OS 32 Represents the optimizer state in FP32 format.

[0054] The specific implementation processes of the forward propagation phase and the back propagation-optimizer update parallel phase described in S3 of the present invention are described in detail below.

[0055] In the forward propagation stage, the GPU updates the model parameters of the large model to be trained layer by layer. In the process of executing the forward calculation of the current network layer, the GPU needs to continuously read the FP16 format model parameters of the subsequent network layers from the SSD through the CPU in advance until the buffer on the GPU is full. At the same time, the GPU needs to unload the calculated gradient checkpoints of the previous layer to the SSD through the CPU; after the forward calculation of each network layer is completed, the GPU needs to immediately release the model parameters of this network layer and start the forward calculation of the next network layer.

[0056] It can be seen that the above forward propagation stage mainly completes the forward propagation in the model training in the heterogeneous training mode. For the convenience of description, the network layer currently being calculated is recorded as network layer i, the next network layer of the forward propagation process is recorded as network layer i+1, and the previous network layer of the forward propagation process is recorded as network layer i-1. The specific process of the above forward propagation stage can be described as follows:

[0057] The FP16 model parameters required for subsequent layer calculations are pre-fetched from the SSD and transferred to the GPU buffer through the CPU buffer for temporary storage. The pre-fetched parameter amount needs to fill the GPU buffer as much as possible. Therefore, when the parameter amount of network layer i+1 is not enough to fill the GPU buffer, it is necessary to further pre-fetch the network parameters of the subsequent network layers. While the network parameters are being communicated and transmitted, the forward calculation of network layer i needs to be completed on the GPU. After the forward calculation of network layer i is completed, the GPU first releases the FP16 format model parameters of network layer i, and then the model parameters of network layer i+1 in the GPU buffer can be directly used to perform the forward calculation of network layer i+1. In addition, during the forward calculation of network layer i+1, the gradient checkpoint of network layer i can be unloaded to the SSD through the CPU in parallel for management.

[0058] In addition, in the back propagation-optimizer update parallel stage, the GPU recalculates and back propagates the large model to be trained layer by layer, while the CPU updates the optimizer state layer by layer; when the GPU is recalculating and back propagating the current network layer, the GPU needs to read the FP16 format model parameters and gradient checkpoints of the subsequent network layers from the SSD in advance through the CPU until the buffer on the GPU is full, and the GPU needs to unload the gradients obtained by the back propagation of the previous network layer to the CPU; after the recalculation and back propagation of each network layer is completed, the GPU needs to immediately release the model parameters and gradient checkpoints of this network layer, and start the recalculation and back propagation of the next network layer; when the CPU is updating the optimizer state of the current network layer, it needs to write the updated optimizer state of the previous layer back to the SSD, and read the optimizer state of the next layer from the SSD in advance and generate a copy of the model parameters in FP32 format, so that the optimizer state update of the next layer can be started immediately after receiving the gradient of the next layer unloaded by the GPU.

[0059] It can be seen that the above reverse-optimizer parallel stage mainly completes the reverse calculation and optimizer calculation of the model in a multi-process in the heterogeneous training mode, and effectively makes the two parallel. Similarly, for the convenience of description, the network layer currently being calculated is recorded as network layer i, but because the back propagation is propagated layer by layer from the reverse side of the network layer, the next network layer of the back propagation process is recorded as network layer i-1, and the previous network layer of the back propagation process is recorded as network layer i+1. The logical process on the GPU and CPU in the above reverse-optimizer parallel stage can be specifically expressed as follows:

[0060] GPU logic: Asynchronously pre-fetch the FP16 model parameters and gradient checkpoints required for subsequent layer calculations from the SSD, and transfer them to the GPU buffer through the CPU buffer for storage. The pre-fetched parameters need to fill the GPU buffer as much as possible. Therefore, when the parameters of network layer i-1 are not enough to fill the GPU buffer, it is necessary to further pre-fetch the network parameters of the subsequent network layers in the reverse propagation direction; at the same time, perform the recalculation of network layer i on the GPU and then perform the reverse propagation of network layer i to obtain the gradient of network layer i. After completing the recalculation and reverse propagation of network layer i, immediately start to use the FP16 format model parameters and gradient checkpoints of network layer i-1 pre-fetched in the buffer to perform the recalculation and reverse propagation of network layer i-1. At the same time, during the recalculation and reverse propagation of network layer i-1, the gradient of network layer i is unloaded to the CPU for updating the optimizer state, and the FP16 model parameters and gradient checkpoints of network layer i are released.

[0061] CPU logic: The CPU subprocess pre-fetches the optimizer state of network layer i from the SSD in advance, and starts executing the optimizer calculation of network layer i immediately after the gradient of network layer i on the GPU is unloaded to the CPU. Then, in parallel with the execution of the optimizer state calculation of network layer i-1, the updated optimizer state of network layer i is written back to the SSD. Therefore, the operation of writing the optimizer state of network layer i back to the SSD is executed in parallel with the update of the optimizer state of network layer i-1.

[0062] It should be noted that, due to the extremely large number of parameters in large models, all intermediate results will not be cached in each iteration in order to save memory. However, gradient calculations often need to access these uncached results, so it is necessary to recalculate them through a recalculation strategy and re-execute part of the forward propagation. The specific method of recalculation belongs to the existing technology and will not be repeated here.

[0063] S4, repeat S3 continuously until all training rounds of the training iteration process are completed. However, before each repetition of S3, it is necessary to analyze whether the buffer size on the CPU and GPU needs to be expanded based on the system information recorded during the previous training process to maximize the training performance.

[0064] It should be noted that the training performance in the present invention refers to the efficiency of training, which can be characterized by the time required to perform a single training (forward + reverse + optimizer update). In an embodiment of the present invention, before each repetition of S3, it is necessary to analyze whether the server's calculation ratio is greater than the communication ratio based on the system information recorded in the last training process. If so, the buffer size on the CPU and GPU is expanded according to the preset increment before repeating S3; after each expansion of the buffer size on the CPU and GPU, it is necessary to determine whether the time required for the next training has changed. If the training time is shorter than before the expansion, the buffer size on the CPU and GPU can be further considered before the next repetition of S3. If the training time is longer than before the expansion, it indicates that the server training performance before the most recent expansion of the buffer has reached its peak, and the buffer can no longer be expanded before the subsequent repetition of S3; and the buffer size on the CPU and GPU can also be considered to be restored to the buffer size corresponding to the peak of the training performance. In addition, in addition to considering that the server's training performance has reached its peak, when expanding the buffer, after any one of the CPU and GPU exceeds the free memory, it is also necessary to no longer expand the buffer. The expansion size of each buffer on the CPU and GPU can be set in advance. Theoretically, the smaller the expansion range, the more likely it is to find the optimal peak. However, considering the efficiency issue, a reasonable expansion range needs to be set, which should not be too small.

[0065] It should be noted that, in the calculation and communication process described in S1 to S4 of the present invention, although only parameters such as gradient, model parameters, checkpoint, and optimizer status are mentioned, based on basic network training knowledge, those skilled in the art should know that some other necessary data in the large model training process also need to be calculated and communicated, such as activation values, etc. However, since the calculation and communication of these data are not the key points of the present invention, they can be performed with reference to the prior art, so the present invention does not describe them in detail.

[0066] It should be noted that the above method steps S1 to S4 can essentially be implemented in the form of a computer program.

[0067] Similarly, based on the same inventive concept, the present invention also provides a computer electronic device corresponding to the single GPU large model training method based on multiple SSDs provided in the above embodiment, which includes a memory and a processor;

[0068] The memory is used to store computer programs;

[0069] The processor is used to implement the single GPU large model training method based on multiple SSDs as described above when executing the computer program;

[0070] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0071] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to a single GPU large model training method based on multiple SSDs, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it can implement the single GPU large model training method based on multiple SSDs as described above.

[0072] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the single GPU large model training method based on multiple SSDs as described above.

[0073] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S4.

[0074] It is understandable that the above storage medium may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which may store program codes.

[0075] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0076] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0077] The training method described in S1~S4 of the present invention unloads all data generated during the training process to the SSD as a whole, including model parameters, optimizer status and gradient checkpoints, so that a model with more than 100B parameters can be trained in a single consumer-grade graphics card and limited CPU memory. At the same time, the present invention reduces the performance overhead caused by the introduction of additional communications in heterogeneous training by improving two aspects. Compared with traditional heterogeneous training, it can be seen that the improvements in these two aspects can be summarized as the hardware and software collaborative design of multiple SSDs and active gradient unloading. The improvement principles and effects are described below to better understand the essence of the present invention.

[0078] 1. Software and hardware collaborative design of multiple SSDs. This improvement mainly solves the problem of decreased training performance due to bandwidth differences between SSD-CPU and CPU-GPU. During the entire training process, the model-related data communicated between the CPU and GPU are FP16 parameters, gradients, and gradient checkpoints, while the model-related data communicated between the SSD and CPU are FP16 parameters and gradients and FP32 parameters, gradients, gradient checkpoints, momentum, variance and other optimizer parameters. The data communicated between the SSD-CPU is 5 times the data communicated between the CPU-GPU, while the bandwidth of the SSD-CPU of a single SSD is only 1 / 3 of the CPU-GPU bandwidth. The present invention combines multiple SSDs in hardware to increase the bandwidth of the SSD-CPU to three times the bandwidth of the CPU-GPU. Although there are some methods that simply increase the number of SSDs that can effectively alleviate the problem of insufficient SSD-CPU bandwidth, the hardware-driven SSD-CPU acceleration is only three times that of the CPU-GPU, while the data flux on the SSD-CPU is five times that of the CPU-GPU, which cannot completely solve the problem of bandwidth differences. Because more SSDs are aggregated from the hardware to increase bandwidth, but there is a lack of software collaborative design, which is mainly reflected in the fact that the communication between SSD, CPU and GPU uses a simple pre-fetch strategy for the next layer. Due to the uneven computing time and communication time of different kernels, it is difficult to parallelize the communication between the three-level memories. At the same time, due to the serious coupling in the design, there is a serious dependence between the communication and computing logics.

[0079] The present invention further optimizes by means of software and hardware collaboration. When communication and calculation are in parallel, an additional buffer space needs to be applied for on the GPU for asynchronous communication. This additional buffer space will affect the overall performance. Specifically, if the buffer is too small, it will lead to insufficient parallelism, and a large amount of communication and calculation will be executed serially. If the buffer is too large, it will lead to insufficient space on the GPU. To determine the size of the buffer, it is first necessary to use the idle amount of CPU and GPU memory in the analysis phase as the upper limit to avoid insufficient CPU and GPU memory due to asynchrony. Secondly, the present invention determines the specific value of the buffer size according to the load ratio of calculation and communication in the analysis phase. The buffer size is set by default to the single largest network layer parameter size. When the calculation ratio is higher, the size of the buffer will be increased. Therefore, the present invention optimizes the optimal buffer on the CPU and GPU through the data of the analysis phase, so that the multi-level communication pipeline is effectively parallelized, hiding the overhead caused by communication without affecting the calculation on the GPU, and improving the overall training performance.

[0080] 2. Active gradient offloading. This improvement mainly solves the problem that the out-of-core optimizer update cannot be effectively parallelized with the reverse calculation on the GPU, resulting in a decrease in training performance. Currently, technologies such as ZeRO-Infinty only offload optimizer calculations to the CPU, while the forward and reverse calculations performed on the GPU are completely serial with the optimizer calculations performed on the CPU. The latter will cause the GPU to be idle and reduce training efficiency. If the forward and reverse calculations on the GPU and the optimizer update on the CPU are simply parallelized, it is not considered that the communication pressure on the SSD-CPU side will cause the CPU-side calculation to become a bottleneck and thus lead to a decrease in training performance. This is mainly manifested in that the SSD-CPU communication and the calculation on the CPU are completely serial, and the independence of resources is not utilized. Secondly, the gradient data is passively offloaded, and there is no coordination between the GPU and the CPU, resulting in the gradient data transmission path being GPU->CPU->SSD and then from SSD->CPU, forming a redundant gradient transmission. However, in the practice of the present invention, the three operations of back propagation on the GPU, optimizer calculation on the CPU, and CPU access to the SSD are completely independent of the resources on the server, which will bring multiple advantages: first, in addition to using multi-process technology to parallelize the reverse calculation on the GPU and the optimizer calculation on the CPU, the optimizer calculation on the CPU and the communication between the SSD-CPU can be further parallelized, which can further reduce the overhead caused by serial communication and calculation; secondly, the active gradient unloading mechanism will actively pre-fetch data from the SSD on the CPU side, and instantly obtain the gradient unloaded by the GPU to complete the optimizer update of the corresponding parameters, which avoids further unloading the gradient to the SSD during passive gradient unloading, and re-reading the gradient from the SSD when the subsequent optimizer update is started, and will not cause the problem of redundant gradient transmission. It can be seen that the active gradient unloading adopted by the present invention reduces the communication pressure and further improves the overall training performance.

[0081] Based on the above theoretical analysis, in order to further demonstrate the impact that the two key improvements of the present invention can have on large-model heterogeneous training, the training method is implemented on specific data through an embodiment.

[0082] Example

[0083] In this embodiment, the heterogeneous training system constructed uses a dual-CPU server, an NVIDIA RTX4090 GPU and 12 SSDs. The GPU and 12 SSDs should be connected to the PCIe slot of the server through the PCIe interface. The training framework of the large model is PyTorch, and the debugging and optimization of the program are implemented through the CUDA Toolkit development toolkit. The specific parameters of the hardware and software are shown in Table 1.

[0084] Table 1 Specific parameters of hardware and software

[0085] In this embodiment, the basic process of the large model training method of the above-mentioned heterogeneous training system is described in S1~S4 above. The process will not be repeated here, and its technical effect will be mainly demonstrated. This embodiment mainly compares the gap between the present invention and the ablation control experiment and other existing heterogeneous training methods (ZeRO-Offload, ZeRO-Infinity and Colossal-AI) from four aspects: maximum trainable model size, end-to-end training performance, multi-SSD software and hardware collaboration and active gradient unloading. The ablation control experiment used for comparison removes the two improvements of multi-SSD software and hardware collaborative design and active gradient unloading compared to the training method shown in S1~S4 above, that is, after the control model removes the software and hardware collaborative design of multiple SSDs, no buffer is set on the CPU and GPU, and the network parameters of the next layer are directly pre-fetched synchronously when the current layer is calculated. At the same time, after the control model removes the active gradient unloading, the gradient on the GPU is no longer unloaded to the CPU, but directly unloaded to the SSD. The CPU re-reads the gradient and the original optimizer state from the SSD before updating the optimizer state of the current network layer. The following shows the performance differences between the method of the present invention and the ablation control experiment in four aspects.

[0086] 1. Maximum trainable model size: The present invention is able to fine-tune models much larger than the benchmark at any CPU and GPU memory capacity, thanks to the ability to fully utilize the memory capacity of the CPU, GPU, and SSD by offloading model states and activations as a whole. With 768 GB of CPU memory, the present invention can fine-tune a 276B model on RTX 4090 and RTX 4080, which are 2.04 times and 3.94 times larger than ZeRO-Infinity, respectively. Even with only 256 GB of CPU memory and RTX 4080, the present invention can successfully train a 175B model.

[0087] 2. End-to-end training performance: When fine-tuning the 13B model on RTX 4090, the training throughput of the present invention is 2.32 times, 3.46 times, and 8.02 times higher than that of ZeRO-Offload, ZeRO-Infinity, and Colossal-AI, respectively. When fine-tuning the 13B model on RTX3090, the training throughput of the present invention is 1.57 times, 2.48 times, and 4.72 times higher than that of ZeRO-Offload, ZeRO-Infinity, and Colossal-AI, respectively.

[0088] 3. Multi-SSD hardware and software collaboration: This embodiment tests the relationship between training throughput and the number of SSDs, such as Figure 4 As shown, the training throughput of the present invention is measured when fine-tuning the 13B model on RTX 4090 based on three BatchSizes of 32, 48 and 64 (respectively denoted as bsz=32, bsz=48, bsz=61), and it is found that a small number of SSDs can approach linear scalability because SSD I / O is the training bottleneck. As the number of SSDs increases from 6 to 12, the throughput gain becomes smaller because the bottleneck of I / O bandwidth shifts to GPU computing and GPU-CPU PCIe transmission. Thanks to the use of software and hardware co-design technology, the gain brought by the present invention by increasing the number of SSDs will be higher than the pure hardware method of increasing the number of SSDs in the ablation control experiment. For the training of the 135B model, the ablation control experiment can provide a 2.82-fold acceleration from 1 SSD to 6 SSDs, while the method of the present invention can provide a 3.42-fold acceleration.

[0089] 4. Active gradient offloading: The active gradient offloading introduced in this invention completely overlaps CPU calculation, GPU calculation and SSD I / O, thereby minimizing the idle time of the GPU. This optimization has a 1.22-fold performance improvement compared to the ablation control experiment without active gradient offloading, and a 1.33-fold performance improvement compared to ZeRO-Infinity.

[0090] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A single GPU large model training method based on multiple SSDs, characterized in that: The training method is implemented based on a server equipped with a single GPU and multiple SSDs; the training method comprises the following steps: S1. On the GPU, the model parameters of the large model to be trained are initialized in FP16 format, and the initialized model parameters are transferred to the SSD for storage; then the model parameters in FP16 format are read from the SSD to the CPU of the server, and the corresponding model parameter copies in FP32 format are initialized on the CPU, and the gradients and optimizer states are initialized based on the model parameter copies in FP32 format, and then the model parameters and optimizer states in FP32 format initialized on the CPU are transferred to the SSD for storage; S2. Perform the first training of the training iteration process on the initialized large model to be trained in a manner that all calculations and communications are performed serially, and record and analyze system information to determine the free memory of the GPU and CPU, the calculation ratio, the communication ratio, and the parameter amount of each of all network layers; then apply for a buffer on the CPU and GPU that does not exceed the free memory, and the buffer size is not less than the parameter amount of the network layer with the largest parameter amount in the large model to be trained; S3, in a way that computation and communication are performed asynchronously, in a two-stage mode consisting of a forward propagation stage and a backpropagation-optimizer update parallel stage, the GPU performs the next training iteration of the training large model to be trained, while recording system information; In the forward propagation stage, the GPU updates the model parameters of the large model to be trained layer by layer, and in the process of executing the forward calculation of the current network layer, the GPU needs to read the FP16 format model parameters of the subsequent network layers from the SSD in advance through the CPU until the buffer on the GPU is full. At the same time, the GPU needs to unload the calculated gradient checkpoints of the previous layer to the SSD through the CPU; after the forward calculation of each network layer is completed, the GPU needs to immediately release the model parameters of this network layer and start the forward calculation of the next network layer; In the back propagation-optimizer update parallel phase, the GPU recalculates and back propagates the large model to be trained layer by layer, while the CPU updates the optimizer state layer by layer; when the GPU is performing the recalculation and back propagation of the current network layer, the GPU needs to read the FP16 format model parameters and gradient checkpoints of the subsequent network layers from the SSD in advance through the CPU until the buffer on the GPU is full, and at the same time, the GPU needs to unload the gradients obtained by the back propagation of the previous network layer to the CPU; after the recalculation and back propagation of each network layer are completed, the GPU needs to immediately release the model parameters and gradient checkpoints of this network layer, and start the recalculation and back propagation of the next network layer; when the CPU is performing the optimizer state update of the current network layer, it needs to write the optimizer state of the previous layer updated back to the SSD, and at the same time read the optimizer state of the next layer from the SSD in advance and generate a copy of the model parameters in FP32 format, so as to start the optimizer state update of the next layer immediately after receiving the gradient of the next layer unloaded by the GPU; S4. Repeat S3 continuously until all training rounds of the training iteration process are completed; and before each repetition of S3, analyze whether it is necessary to expand the buffer size on the CPU and GPU according to the system information recorded in the previous training process, so as to maximize the training performance.

2. The single GPU large model training method based on multiple SSDs as claimed in claim 1, characterized in that: The GPU and SSD are connected to the PCIe slot of the server through the PCIe interface, and all SSDs receive data transmitted by the GPU and CPU according to the storage strategy of the server and perform persistent storage.

3. The single GPU large model training method based on multiple SSDs as claimed in claim 1, characterized in that: The server records system information during each round of training through the PyTorch Hook mechanism, including the free memory on the GPU and CPU, the communication bandwidth between the CPU and GPU, the computing time required for forward calculation of each network layer on the GPU, the computing time required for reverse calculation of each network layer on the GPU, and the parameter amount of each network layer in the large model to be trained.

4. The single GPU large model training method based on multiple SSDs as claimed in claim 1, characterized in that: Before each repetition of S3, it is necessary to analyze whether the server's computing ratio is greater than the communication ratio based on the system information recorded during the previous training process. If so, the buffer size on the CPU and GPU is increased according to the preset increment before repeating S3; the buffer will no longer be increased after the server's training performance reaches its peak or the free memory of either the CPU or GPU exceeds the amount.

5. The single GPU large model training method based on multiple SSDs as claimed in claim 1, characterized in that: The GPU is a consumer-grade GPU.

6. The single GPU large model training method based on multiple SSDs as claimed in claim 1, characterized in that: The GPU does not persistently store any data, but only saves temporary data required for forward and reverse calculations; the CPU does not persistently store any data, but only saves temporary data required for optimizer updates; the SSD persistently stores various types of data required for model training, including model parameters in FP16 format and optimizer status data.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by the processor, it can implement the single GPU large model training method based on multiple SSDs as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the single GPU large model training method based on multiple SSDs as described in any one of claims 1 to 6 is implemented.

9. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the single GPU large model training method based on multiple SSDs as described in any one of claims 1 to 6 when executing the computer program.

10. A single GPU large model training system based on multiple SSDs, characterized in that: It includes a server and a single GPU and multiple SSDs installed on the server; a large model training control program runs on the server, and the large model training program is used to implement the single GPU large model training method based on multiple SSDs as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Method and system for deep learning based on GPU

    CN105224502A

  • Large-scale deep learning method and system based on solid-state disk extended memory

    CN110032450A

  • Synchronous training method and device, computer device and storage medium

    CN110688230A

  • Visual large model distributed training method and system

    CN113723443A

  • Scheduling method and device for model training, electronic equipment and storage medium

    CN114911596A

Cited By

  • Graph neural network training acceleration method and system based on multiple GPUs and multiple SSDs

    CN119129683A

  • Large model training fault tolerance method and system, medium and program product

    CN119938407A

  • A Fault Tolerance Method, System, Medium and Program Product for Large Model Training

    CN119938407B

  • DeepSpeed-based gradient unloading optimization method and device and storage medium

    CN120144304A

  • Large model distributed parallel training method, program product, computing node and storage medium

    CN121412032A