A heterogeneous space optimizer based on tensor asynchronous hard disk read-write

Abstract

The application discloses a heterogeneous space optimizer based on tensor asynchronous hard disk reading and writing, relates to the technical field of hard disk reading and writing and comprises an AdamOptimizer mainstream optimizer, an SGDOptimizer mainstream optimizer, an NVMeOptimizer base class optimizer and a TensorNVMe asynchronous reading and writing library. The AdamOptimizer mainstream optimizer is a current mainstream optimizer. The SGDOptimizer mainstream optimizer is a current mainstream optimizer. The NVMeOptimizer base class optimizer overlaps the calculation in the optimization process and the hard disk reading and writing by calling the TensorNVMe asynchronous reading and writing library. The heterogeneous space optimizer based on tensor asynchronous hard disk reading and writing is provided with the NVMeOptimizer base class optimizer, so that the NVMeOptimizer base class optimizer overlaps the calculation in the optimization process and the hard disk reading and writing by calling the TensorNVMe asynchronous reading and writing library, and the running efficiency is greatly improved. The heterogeneous space optimizer inherits the NVMeOptimizer base class optimizer, and only needs to be simply modified to be compatible with the current mainstream optimizer.

Description

Technical Field

[0001] This invention relates to the field of hard disk read / write technology, specifically to a heterogeneous space optimizer based on tensor asynchronous hard disk read / write. Background Technology

[0002] The emergence of pre-trained models (PTMs), represented by BERT and GPT, is a milestone in the field of Natural Language Processing (NLP). NLP is entering the pre-training era. PTMs use neural networks with stacked Transformer structures to pre-train general language feature representations on large amounts of text, and then transfer the learned knowledge to different downstream tasks through fine-tuning. Using massive amounts of text data from the Internet, PTMs can capture subtle features of natural language and enable downstream tasks to achieve more impressive performance. Therefore, the consensus in the AI ​​community is to adopt PTMs as the backbone for specific NLP tasks, rather than training models from scratch on task-specific datasets.

[0003] The power of PTM lies in its hundreds of millions of parameters, which places extremely high demands on the computational and memory resources required to run it. Because PTM model data cannot fit into the memory of a single GPU, the data parallelism techniques used by most deep learning training frameworks are unsuitable for PTM. The limited storage capacity of GPU hardware acts as a wall restricting the trainable scale of PTM, a phenomenon often referred to as the "GPU memory wall." In the past two years, the industry has attempted to overcome the GPU memory wall limitation by distributing model data across multiple GPUs using parallel training, such as ZeRO-DP, model parallelism, and pipeline parallelism. However, these techniques still heavily rely on continuously expanding GPU scale, while heterogeneous training techniques have emerged as a game-changer. It can not only significantly increase the scale of single-GPU trained models but also be used orthogonally with parallel training techniques. We divide each training step of a neural network into three stages: forward propagation, back propagation, and optimizer optimization.

[0004] Microsoft's DeepSpeed ​​pioneered a heterogeneous space optimizer based on tensor-synchronous disk read / write. It achieves heterogeneous training by synchronously offloading the optimizer state to disk. Specifically, during forward and backward propagation, the optimizer state is stored on disk; during optimization, the optimizer state is read from disk, the corresponding calculations are performed, and then it is offloaded back to disk. Using single-precision floating-point training mode and the Adam optimizer, with N model parameters, the required memory is 4N (parameters) + 4N (gradients) + 8N (optimizer state) = 16N. By offloading the optimizer state to disk, the required memory can be halved, thus overcoming the memory wall limitation.

[0005] Because the optimizer's optimization process is independent for each parameter and optimizer state, sequential optimization and concurrent optimization of parameters do not affect correctness. Furthermore, the resources required for disk read / write and computation are relatively orthogonal, meaning disk read / write and computation can be performed simultaneously. DeepSpeed ​​performs read, computation, and unloading sequentially, without overlapping disk read / write and computation, resulting in low actual operating efficiency. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a heterogeneous space optimizer based on tensor asynchronous hard disk read / write, which solves the problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: a heterogeneous space optimizer based on tensor asynchronous hard disk read and write, including the AdamOptimizer mainstream optimizer, the SGDOptimizer mainstream optimizer, the NVMeOptimizer base class optimizer, and the TensorNVMe asynchronous read and write library;

[0008] The AdamOptimizer is a mainstream optimizer, and it is one of the existing mainstream optimizers.

[0009] The SGDOptimizer is a mainstream optimizer, and is one of the existing mainstream optimizers.

[0010] The NVMeOptimizer base class optimizer overlaps the computation and disk read / write operations during the optimization process by calling the TensorNVMe asynchronous read / write library.

[0011] The TensorNVMe asynchronous read / write library can asynchronously read tensors from the hard disk into memory or unload them from memory to the hard disk.

[0012] Optionally, the heterogeneous space optimizer can be modified by inheriting from NVMeOptimizer to be compatible with existing mainstream optimizers.

[0013] Optionally, a heterogeneous space optimization method based on tensor asynchronous hard disk read / write includes the following steps:

[0014] S1: Store the optimizer state on the hard disk;

[0015] S2: During optimization, it is dynamically read into memory and then calculated together with the parameters;

[0016] S3: After the calculation is complete, unload it back to the hard drive.

[0017] Optionally, in step S2, during optimization, the optimizer process is independent for each parameter and optimizer state. Sequential optimization of each parameter and concurrent optimization of each parameter do not affect the correctness.

[0018] Optionally, during optimization, S2 dynamically reads the data into memory and then performs calculations together with the parameters. The resources required for the disk read / write and calculation tasks are relatively orthogonal, and disk read / write and calculation can be performed simultaneously.

[0019] Optionally, during optimization, S2 dynamically reads the tensor into memory and then calculates it together with the parameters. The tensor reading and writing on the hard disk is divided into two stages: submitting the task and waiting for the task to complete.

[0020] Optionally, the steps of storing the optimizer state on the hard disk in S1, dynamically reading it into memory during optimization in S2 and then calculating it together with the parameters in S2, and unloading it back to the hard disk after the calculation is completed in S3, overlap the calculation and hard disk read / write through asynchronous tensor read / write.

[0021] This invention provides a heterogeneous space optimizer based on tensor asynchronous hard disk read / write, which has the following beneficial effects:

[0022] 1. This heterogeneous space optimizer based on tensor asynchronous disk read / write uses the NVMeOptimizer base class optimizer. By calling the TensorNVMe asynchronous read / write library, the NVMeOptimizer base class optimizer overlaps the computation and disk read / write in the optimization process, thereby greatly improving the running efficiency. This heterogeneous space optimizer inherits from the NVMeOptimizer base class optimizer and can be compatible with existing mainstream optimizers with only simple modifications.

[0023] 2. This heterogeneous space optimizer based on tensor asynchronous hard disk read and write improves the optimizer's running efficiency by overlapping hard disk read and write with computation through the tensor asynchronous hard disk read and write method. It further breaks through the memory wall limitation of existing hardware, greatly improves the availability of large AI models, and significantly reduces the hardware requirements and economic burden when training large AI models. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0025] Figure 2 This is a flowchart of the optimizer optimization process of the present invention;

[0026] Figure 3This is a diagram of the heterogeneous space optimization method of the present invention.

[0027] In the diagram: 1. AdamOptimizer, a mainstream optimizer; 2. SGDOptimizer, a mainstream optimizer; 3. NVMeOptimizer, a base class optimizer; 4. TensorNVMe asynchronous read / write library. Detailed Implementation

[0028] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0029] Please see Figures 1 to 3 The present invention provides a technical solution: a heterogeneous space optimizer based on tensor asynchronous hard disk read and write, including AdamOptimizer mainstream optimizer 1, SGDOptimizer mainstream optimizer 2, NVMeOptimizer base class optimizer 3 and TensorNVMe asynchronous read and write library 4;

[0030] AdamOptimizer is a mainstream optimizer; it is one of the existing mainstream optimizers.

[0031] SGDOptimizer is a mainstream optimizer, and one of the existing mainstream optimizers.

[0032] The NVMeOptimizer base class optimizer 3 overlaps the computation and disk read / write operations during the optimization process by calling the TensorNVMe asynchronous read / write library 4.

[0033] TensorNVMe asynchronous read / write library 4 is capable of asynchronously reading tensors from disk into memory or unloading tensors from memory into disk.

[0034] Furthermore, the heterogeneous space optimizer can be modified by inheriting from NVMeOptimizer to be compatible with existing mainstream optimizers, including not only Adam and SGD.

[0035] A heterogeneous space optimization method based on tensor asynchronous hard disk read / write includes the following steps:

[0036] S1: Store the optimizer state on the hard disk;

[0037] S2: During optimization, it is dynamically read into memory and then calculated together with the parameters;

[0038] S3: After the calculation is complete, unload it back to the hard drive.

[0039] Furthermore, during optimization, S2 dynamically reads the parameters into memory and then calculates them together with the parameters. The optimization process is independent for each parameter and the optimizer state. The sequential optimization of each parameter and the concurrent optimization of each parameter do not affect the correctness, thus ensuring that the correctness of the parameters is not affected.

[0040] Furthermore, during optimization, S2 dynamically reads the data into memory and then performs calculations along with the parameters. The resources required for disk read / write and calculation tasks are relatively orthogonal, allowing disk read / write and calculation to be performed simultaneously. This overlaps disk read / write and calculation, improving the optimizer's operating efficiency and further breaking through the memory wall limitations of existing hardware.

[0041] Furthermore, during optimization, S2 dynamically reads the data into memory and then performs calculations on it along with the parameters. The process of reading and writing tensors on the hard disk is divided into two stages: submitting the task and waiting for the task to complete.

[0042] Furthermore, by storing the optimizer state on the hard disk in step S1, dynamically reading it into memory during optimization in step S2 and then calculating it together with the parameters in step S3, and unloading it back to the hard disk after the calculation is completed, the calculation and hard disk read / write are overlapped through asynchronous tensor read / write, which greatly improves the availability of large AI models and significantly reduces the hardware requirements and economic burden when training large AI models.

[0043] In summary, this heterogeneous space optimizer based on tensor asynchronous disk read / write operates by storing the optimizer state on the hard disk and dynamically loading it into memory during optimization. This state is then used in conjunction with parameters for calculation. After calculation, the optimized state is unloaded back onto the hard disk. Through asynchronous tensor read / write, the calculation and hard disk read / write operations are overlapped. The optimization process is as follows: Figure 2 As shown:

[0044] When calculating optimizer state / parameter 1, it can simultaneously read optimizer state 2; when calculating optimizer state / parameter 2, it can simultaneously read optimizer state 3 and unload optimizer state 1.

[0045] By using tensor asynchronous disk read / write technology, disk read / write and computation are overlapped, improving the optimizer's operating efficiency, further breaking through the memory wall limitation of existing hardware, greatly improving the availability of large AI models, and significantly reducing the hardware requirements and economic burden when training large AI models.

[0046] Example

[0047] By training GPT2-XL with 1.5 billion parameters and GPT2-4B with 4 billion parameters respectively, this invention achieves speedups of 30.8% and 26.7% compared to the existing DeepSpeed, respectively. Simultaneously, hard drive read / write speed tests were conducted, bandwidth was calculated, and the theoretical read / write time was deduced based on the optimizer state. This allowed monitoring of the optimizer's runtime, revealing that the runtime was very close to the theoretical read / write time required by the optimizer state. Since hard drive read / write is typically much slower than computation, this result indicates that this invention has largely and successfully overlapped computation and hard drive read / write operations.

[0048] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A heterogeneous space optimizer based on tensor asynchronous hard disk read-write, characterized in that: The AdamOptimizer mainstream optimizer (1), the SGDOptimizer mainstream optimizer (2), the NVMeOptimizer base class optimizer (3) and the TensorNVMe asynchronous read-write library (4); The NVMeOptimizer base class optimizer (3) overlaps the calculation and hard disk read-write in the optimization process by calling the TensorNVMe asynchronous read-write library (4); The TensorNVMe asynchronous read-write library (4) can asynchronously read tensors from the hard disk into the memory or unload tensors from the memory to the hard disk; The heterogeneous space optimizer can be compatible with the AdamOptimizer mainstream optimizer (1) and the SGDOptimizer mainstream optimizer (2) by inheriting the NVMeOptimizer and modifying; The heterogeneous space optimizer based on the heterogeneous space optimization method of tensor asynchronous hard disk read-write, characterized in that it comprises the following steps: S1: store the optimizer state in the hard disk; S2: during optimization, dynamically read it into the memory, and then calculate it together with the parameters; S3: after the calculation is completed, unload it back to the hard disk; The optimization process of the optimizer in the step of dynamically reading it into the memory during optimization and then calculating it together with the parameters is independent for each parameter and optimizer state, and sequentially optimizing each parameter and concurrently optimizing each parameter does not affect the correctness; In the step of dynamically reading it into the memory during optimization and then calculating it together with the parameters, the resources required for hard disk read-write and calculation are relatively orthogonal, and hard disk read-write and calculation can be performed simultaneously; In the step of dynamically reading it into the memory during optimization and then calculating it together with the parameters, the read-write of tensors on the hard disk is divided into two stages: submitting a task and waiting for the task to be completed; Through the steps of storing the optimizer state in the hard disk S1, dynamically reading it into the memory during optimization S2, and then calculating it together with the parameters, and the step of unloading it back to the hard disk after the calculation is completed S3, the calculation and hard disk read-write are overlapped through the asynchronous read-write of tensors.

Images