Method, system and device for efficient memory replacement between GPU devices and storage medium

Abstract

The invention discloses a method, system and device for efficient memory replacement between GPU devices and a storage medium. In a related scheme, firstly, on the premise of effectively reducing memory limitation, data exchange operation (unloading and retrieving) is parallel to model training, calculation overhead is not introduced, and transmission time can be hidden; and secondly, the inactive data on the GPU equipment with high memory load are unloaded to other GPU equipment and are retrieved when needed, so that the free memory space of the equipment in the system is fully utilized, a plurality of direct connection high-speed links among the GPUs are aggregated, and the high-speed communication bandwidth is obtained, so that the memory is reduced more quickly, and the data are retrieved more timely. By combining the two points, the performance overhead introduced by memory compression can be greatly reduced, and the limitation of the memory on model training can be effectively reduced, so that the model training efficiency is improved.

Description

technical field [0001] The present invention relates to the technical field of GPU device memory replacement, and in particular, to a method, system, device and storage medium for efficient memory replacement between GPU devices. Background technique [0002] In 2020, the deep learning team of New York University in the United States published the deep learning parallel training memory compression system SwapAdvisor (SwapAdvisor: Push Deep Learning Beyond the GPUMemory Limitvia Smart Swapping) at the ASPLOS (Architectural Support for Programming Languages ​​and Operating Systems) conference. Swap to CPU (central processing unit) memory, and then swap from CPU memory to GPU (graphics processing unit) memory next time you need it. However, the bandwidth of data replacement between CPU and GPU through PCIe Gen3x16 link is 16GB / s, which is only 60% of the bandwidth of NVLink2.0 of a direct link between GPUs. As a result, on DGX-1, the upper limit of replacement data speed is onl...

AI Technical Summary

Problems solved by technology

In addition, in the parallel training scenario, in order to realize the data replacement between each GPU and the CPU in the system, the CPU memory becomes the load bottleneck. On DGX-1, the number of links between the CPU and the shared PCIe is 4:1, and the data replacement cost varies with The number of GPUs increases linearly; especially for the pipeline parallel training scenario, the pipeline stage is designed to balance the calculation load, but the memory requirements of each stage of this pipeline are unbalanced. Compared with the tail, the head of the pipeline The department must store more intermediate data for reverse calculation, resulting in ResNet152 model training with a batch size greater than 128 and Bert model training with a parameter volume greater than 640 million on a server with 8 NVIDIA 1080Ti GPU devices There was an error of insufficient memory, and the memory consumption of the entire model at this time only accounted for 68.9% and 88.7% of the total memory of the 8 GPUs. Therefore, it is necessary to develop a new memory replacement technology to improve data exchange performance

Images