Large-scale multi-machine multi-card pre-training method, system, equipment and server cluster

By using slurm and Horovod frameworks in a multi-machine and multi-card environment, combining IB/RDMA communication and Apex hybrid accuracy, the problem of large-scale multi-machine and multi-card computing and hybrid heterogeneous machines is solved, and efficient model training and Resource utilization improves model accuracy and training speed.

CN113723552BActive Publication Date: 2024-11-08PEKING UNIV SHENZHEN GRADUATE SCHOOL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111042840.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-07
Publication Date
2024-11-08
Estimated Expiration
2041-09-07

AI Technical Summary

Technical Problem

The prior art is difficult to realize large-scale multi-machine and multi-card computing and effectively use hybrid heterogeneous machines for parallel training, especially in the case of super-large data sets and large batch sizes, which have problems such as decreasing model accuracy and low resource utilization.

Method used

Using a multi-machine and multi-card parallel training method based on the slurm framework and Horovod framework, combined with IB/RDMA communication, Apex hybrid accuracy and appropriate optimizers, we realize multi-machine and multi-card pre-training for isomorphic and heterogeneous hybrid models to optimize the CPU Core allocation and data storage accelerate the training process.

Benefits of technology

It significantly shortens the training time of large-scale model, improves model accuracy and resource utilization, expands the scale of parallel training clusters, and provides an implementation idea for large-scale model training on hybrid heterogeneous platforms.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113723552B_ABST
    Figure CN113723552B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of distributed training technology, and discloses a large-scale multi-machine multi-card pre-training method, system, device and server cluster. Multi-machine multi-cards are deployed on multiple servers to perform multi-machine multi-card parallel training of homogeneous and heterogeneous hybrid models; large-scale multi-machine multi-card training and evaluation are performed based on the slurm framework, and the unsupervised feature learning BYOL algorithm is used as an example for implementation; large-scale multi-machine multi-card training and evaluation are performed based on the Horovod framework, and the video semantic unsupervised learning PRP algorithm is used for implementation; the training includes environment configuration, task configuration, communication configuration, and task acceleration. The multi-machine multi-card large-scale training experiment of the present invention has a high batch size and a short training time compression, which verifies the parallel capability of the Pengcheng Cloud Brain I large scientific device and expands the cluster scale of parallel training, which has guiding significance for using ultra-large-scale clusters to carry out distributed training.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of distributed training technology, and in particular relates to a large-scale multi-machine multi-card pre-training method, system, equipment and server cluster. Background Art

[0002] At present, in response to my country's demand for the construction of an open source, open and shared innovation platform for AI, Pengcheng Laboratory has launched the Pengcheng Cloud Brain Phase I platform. Pengcheng Cloud Brain I is a large cluster system built with NVIDIA GPU servers as the infrastructure. As an AI large scientific device to support the construction of a better AI ecosystem, Pengcheng Cloud Brain I has cluster management tools and resource scheduling platforms, and supports running AI tasks in GPU clusters. In the process of building and upgrading smart cities, the amount of data has increased dramatically, and as artificial intelligence tasks become more complex and diverse, the scale of models has also increased. At present, there are many practical applications that face the need to use large-scale data to train large models. Using multiple machines and multiple cards to carry out distributed training is a necessary way to meet such needs. Therefore, large-scale multi-machine and multi-card distributed training based on Pengcheng Cloud Brain I can significantly improve the efficiency of model training.

[0003] In terms of the current scale of resources used in distributed training, the BYOL algorithm reproduced by OpenMMLab has public data showing that only 128 GPU cards were used for testing, and the maximum batch size was 4096. Currently, few units at home and abroad can complete large-scale multi-machine and multi-card operations with powerful computing power, and there is a problem of reduced model accuracy when training large batch sizes of large data sets. In addition, how to effectively use hybrid heterogeneous machines for parallel training is also a difficulty in the field of parallel computing, which is of great significance for practical applications. Therefore, a new large-scale multi-machine and multi-card pre-training method is urgently needed.

[0004] Through the above analysis, the problems and defects of the existing technology are as follows: At present, in the multi-machine and multi-card data training, few units of the existing technology can complete large-scale multi-machine and multi-card operations with powerful computing power, and there is a problem of reduced model accuracy when training with ultra-large data sets and large batch sizes; at the same time, how to effectively use hybrid heterogeneous machines for parallel training is also a difficulty in the field of parallel data computing.

[0005] The difficulty of solving the above problems and defects is: communication bottlenecks and abnormal monitoring problems during large-scale multi-machine and multi-card training, how to select appropriate parameter adjustment strategies during large batch size training to make the model converge and improve accuracy, and ensure stable operation while ensuring algorithm performance. In addition, with the development of technology and different needs, it is not guaranteed that all machines in the resource pool are of the same type. Different types of machines have different configurations. How to effectively use mixed heterogeneous machines for parallel training.

[0006] The significance of solving the above problems and defects is as follows: the use of larger-scale clusters and successful model training greatly shortens the model training time and improves the model accuracy. It provides a technical approach for using ultra-large-scale data training and using general large models, which can support more downstream tasks while ensuring algorithm performance. Parallel training of mixed heterogeneous machines can improve resource utilization and further increase the scale of parallel training. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a large-scale multi-machine and multi-card pre-training method, system, device and server cluster, and more particularly, relates to a large-scale multi-machine and multi-card (GPU) pre-training method, system, device and server cluster based on Pengcheng Cloud Brain I.

[0008] The present invention is implemented as follows: a large-scale multi-machine multi-card pre-training method, comprising:

[0009] Deploy multiple machines and multiple cards on multiple servers to run multiple machines and multiple cards in parallel on homogeneous and heterogeneous mixed models;

[0010] Based on the slurm framework, large-scale multi-machine and multi-card training and evaluation are carried out, taking the unsupervised feature learning BYOL algorithm as an example;

[0011] Large-scale multi-machine and multi-card training and evaluation based on the Horovod framework, implemented with the video semantic unsupervised learning PRP algorithm;

[0012] The training includes environment configuration, task configuration, communication configuration, task acceleration, etc.

[0013] Specifically include:

[0014] Step 1: Unsupervised feature learning: Use the BYOL algorithm to deploy multiple machines and multiple cards;

[0015] Step 2: Unsupervised learning of video semantics: Use the PRP algorithm to deploy multiple machines and multiple cards;

[0016] Step 3: Multi-machine and multi-card pre-training: Perform multi-machine and multi-card mixed model training.

[0017] Furthermore, in step one, further comprising:

[0018] Using N DGX2s with a total of 16×N V100s, the Imagenet2012 dataset is trained using the unsupervised feature learning BYOL algorithm to obtain a pre-trained model, and the training time is compressed from 7 days to 5 hours and 6 minutes. The solution includes: setting a CPU server as the master node, other GPU servers as computing nodes, and the master node is shared with one of the GPU servers; the master node and each child node submit corresponding deployment scripts.

[0019] Furthermore, in step one, further comprising:

[0020] (1) Multi-machine deployment based on slurm on Cloud Brain I. Since the control node, i.e. the master node, does not participate in the calculation, only the CPU needs to be applied for the control node;

[0021] If the server model is DGX2, the configuration of the node parameters includes:

[0022] 1) Configure the operating environment for each node in a mirrored manner;

[0023] 2) The configuration of the control node is: the control node task does not apply for GPU, the number of CPU cores applies for 6 cores, the memory applies for 100G, and it is set as the backbone task; the configuration of the sub-nodes is: each sub-node task applies for 16 GPUs, the number of CPU cores applies for 80 cores, and the memory applies for 1T; according to the control node and sub-nodes sharing a server, the total number of machines used is equal to the number of sub-nodes;

[0024] 3) Both the control node and the sub-nodes use IB / RDMA for multi-machine communication and configure half of the memory as shared memory;

[0025] 4) Configure the startup command and training script to start running parallel training tasks.

[0026] (2) Configure a multi-machine slurm environment based on debug mode.

[0027] Debug the DGX2 machine through debug mode: enter the task executed on DGX2 through SSH; if slurm-related software is not installed in the environment, install it; if slurm-related software has been pre-installed, verify it through slurmd-V and slurmd-C commands, and check the number of CPU cores; in the cloud brain version of slurm multi-machine deployment, modify masterip.txt and slaveip.txt, add the master node, that is, the IP address of the control node, to masterip.txt; add the child node, that is, the IP address of the computing node, to slaveip.txt, and modify the ControlMachine variable in the slurm_autoconfig.sh script, and then execute the bash slurm_autoconfig.sh script to complete the configuration of the multi-machine slurm environment.

[0028] (3) Start Cloud Brain I high-speed multi-machine communication IB / RDMA

[0029] 1) Choose Nvidia NGC19.10 as the base image. The image link is: https: / / docs.nvidia.com / deeplearning / frameworks / pytorch-release-notes / rel_19-10.html#rel_19-10;

[0030] 2) Specify the IB network card in the training script:

[0031] ①os.environ['NCCL_IB_HCA']="mlx5_0";

[0032] ②os.environ['NCCL_DEBUG']="INFO".

[0033] (4) Acceleration solutions for large-scale tasks

[0034] The following measures are taken to speed up the training process:

[0035] 1) Dataset storage acceleration: A dedicated dataset storage space is created and mounted in memory, so that the dataset can be used continuously without restarting the system. This accelerates data reading and training.

[0036] 2) Using IB / RDMA for multi-machine communication, the training speed can be improved by accelerating the data interaction process between multiple machines in training;

[0037] 3) Using Apex mixed precision for training reduces the video memory usage by half compared to single-precision floating point training, so the batch size can also be doubled;

[0038] 4) Use optimizers suitable for large-scale batch sizes, such as lars, lambda, yogi, etc., to increase the training speed through optimizer acceleration;

[0039] 5) The allocation of CPU cores is optimized for slurm. When the total number of cores is fixed, the efficiency of CPU allocation job is improved as much as possible.

[0040] (5) Based on steps (1) to (4), the multi-machine and multi-card parallel version of the BYOL algorithm under the slurm framework is used to perform large-scale multi-machine and multi-card model pre-training.

[0041] (6) Use a single machine with multiple cards to evaluate the pre-trained model of the BYOL algorithm.

[0042] The author of the BYOL algorithm used resnet50 as the base network and trained for 200 rounds with a total batchsize of 4096 to obtain a pre-trained model. Based on the pre-trained model, a single machine with 8 graphics cards and a total batchsize of 256 was used to evaluate the ImageNet LinearClassification task. In the 100 rounds of training for the evaluation task, the highest top1-accuracy on the validation set was 67.10.

[0043] The pre-trained model used in the evaluation is: a pre-trained model obtained by using 8 DGX2s with a total of 128 V100s, a total batchsize of 12288, and 200 rounds of training with the base network resnet101; based on the pre-trained model, a single machine with 16 cards and a total batchsize of 2048 is used to evaluate the ImageNet Linear Classification task. In the 100 rounds of training for the evaluation task, the highest top1-accuracy on the validation set is 69.294.

[0044] Further, in step 2, the video semantic unsupervised learning includes:

[0045] (1) Deploy the Horovod framework to Pengcheng Cloud Brain I.

[0046] Use the image to install and deploy the software environment required by Horovod on the node, and set up ssh password-free login; when starting the task, select the image of the software required by Horovod that has been installed in Cloud Brain I; add the execution of the ssh login script to the task startup command to complete multi-machine ssh password-free login.

[0047] (2) From the deployment of the environment to the start of training, the intermediate steps are the same as above, except that the master node can apply for a GPU as a computing node at the same time. The node parameter configuration, startup of Cloud Brain I high-speed multi-machine communication IB / RDMA and the acceleration solution for large-scale tasks are the same as above.

[0048] (3) The PRP algorithm is changed to a multi-machine and multi-card parallel version under the Horovod framework. Based on steps (1) to (2), large-scale multi-machine and multi-card model pre-training is carried out.

[0049] Furthermore, in step (1), the multi-machine password-free configuration script includes:

[0050] First generate the secret key, including the private key and the public key, copy the secret key to the shared storage directory of the Cloud Brain platform, and create a shell script for ssh login in this directory. The script content is to distribute the secret key to the .ssh path of the root directory of each node and modify the corresponding permissions; this step is convenient for later debugging through the Cloud Brain debug mode, without password login to the machine where each task is located.

[0051] Furthermore, in step 3, the multi-machine multi-card pre-training includes:

[0052] On the Cloud Brain I platform, a multi-queue application method is used to implement mixed training of cards of different models. The source file is modified and remounted. The pod_id of the source file is obtained through scripting, and the IP and host name of each queue are improved. After remounting, the / etc / hosts in each queue will have the IP and host name of all queues to ensure communication between machines.

[0053] Another object of the present invention is to provide a large-scale multi-machine multi-card pre-training system using the large-scale multi-machine multi-card pre-training method, wherein the large-scale multi-machine multi-card pre-training system comprises:

[0054] Unsupervised feature learning module for multi-machine and multi-card deployment using BYOL algorithm;

[0055] Video semantic unsupervised learning module, used for multi-machine and multi-card deployment using the PRP algorithm;

[0056] The multi-machine and multi-card pre-training module is used for multi-machine and multi-card mixed model training.

[0057] Another object of the present invention is to provide a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0058] (1) Unsupervised feature learning: Use BYOL algorithm for multi-machine and multi-card deployment;

[0059] (2) Unsupervised learning of video semantics: using the PRP algorithm for multi-machine and multi-card deployment;

[0060] (3) Multi-machine and multi-card pre-training: Perform multi-machine and multi-card mixed model training.

[0061] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the following steps:

[0062] (1) Unsupervised feature learning: Use BYOL algorithm for multi-machine and multi-card deployment;

[0063] (2) Unsupervised learning of video semantics: using the PRP algorithm for multi-machine and multi-card deployment;

[0064] (3) Multi-machine and multi-card pre-training: Perform multi-machine and multi-card mixed model training.

[0065] Another object of the present invention is to provide an information data processing server cluster, which is used to implement the large-scale multi-machine and multi-card pre-training system.

[0066] Combined with all the above technical solutions, the advantages and positive effects of the present invention are as follows: the large-scale multi-machine and multi-card pre-training method provided by the present invention uses 25 DGX2s with a total of 400 V100s to complete the test, and the maximum batch size is 76800. At present, few units at home and abroad can complete large-scale multi-machine and multi-card operations with such powerful computing power. The multi-machine and multi-card large-scale training experiment involved in the present invention has a high batch size and a short training time compression. It not only verifies the parallel capability of the Pengcheng Cloud Brain I large scientific device, but also further expands the cluster scale of parallel training. It has great guiding significance for the feasibility and specific implementation methods of distributed training using ultra-large-scale clusters. At the same time, the model accuracy also shows good performance in the downstream task evaluation after pre-training. The heterogeneous hybrid machine model training scheme involved provides an implementation idea for large-scale model training based on hybrid heterogeneous platforms.

[0067] The implementation of the unsupervised feature learning application example provided by the present invention is as follows:

[0068] (1) The benchmarking work is to use resnet50 as the base network and a total batch size of 4096 to train for 200 rounds to obtain a pre-trained model, and then evaluate the ImageNetLinearClassification task based on the pre-trained model. In the evaluation task, a total of 100 rounds of training are performed, using a single machine with 8 cards and a total batch size of 256. The highest top1 accuracy obtained on the validation set is 67.10;

[0069] (2) The present invention uses resnet101 as the base network and a total batch size of 12288 for 200 rounds of training to obtain a pre-trained model, and evaluates the ImageNetLinearClassification task based on the pre-trained model. In the evaluation task, a total of 100 rounds of training are performed, and a single machine with 16 cards and a total batch size of 2048 are used for testing. The highest top1 accuracy obtained on the validation set is 69.294;

[0070] (3) Using the same parameters as the author, the highest top1 accuracy obtained by the present invention on the validation set is 67.068, while the highest data published by the author is 67.10, which are basically consistent.

[0071] (4) For the application of unsupervised learning of video semantics, since there is no multi-machine multi-card program for the PRP algorithm, the original single-machine multi-card program was modified to complete multi-machine multi-card training and testing.

[0072] Based on the Pengcheng Cloud Brain I platform, the present invention proposes and implements a large-scale multi-machine multi-card distributed training method, and successfully realizes multi-machine multi-card parallelism of homogeneous and heterogeneous hybrid models. Based on the slurm framework and the Horovod framework, the entire process of large-scale multi-machine multi-card training and evaluation of the BYOL algorithm and the PRP algorithm is implemented, including environment configuration, task configuration, communication configuration, task acceleration, etc., which greatly shortens the training time of large models on large data sets, and also further expands the scale of machines involved in training. The training time compression of the two sets of multi-machine multi-card large-scale computing algorithms reaches the expected goal.

[0073] The completion status of multiple machines and multiple cards of the same model in parallel is as follows:

[0074] (1) We used 25 DGX2s with a total of 400 V100s, deployed the slurm framework, configured a multi-machine parallel environment, combined multiple acceleration strategies, and adopted the unsupervised feature learning BYOL algorithm to train the Imagenet2012 dataset and obtain a pre-trained model, which reduced the training time from 7 days to 5 hours and 6 minutes, achieving the expected goal;

[0075] (2) We used 14 DGX2s with a total of 224 V100s, deployed the Horovod framework, configured a multi-machine parallel environment, combined multiple acceleration strategies, and adopted the video semantic unsupervised learning PRP algorithm to train the Kinetics400 dataset, shortening the training time from 10 days to 11 hours and 42 minutes, achieving the expected goal.

[0076] (3) For mixed models, such as DGX1, DGX2, AGX and other models on Cloud Brain I, mixed training of multiple models of the above two algorithms is implemented.

[0077] The key points of the present invention are:

[0078] (1) Based on the Pengcheng Cloud Brain I platform, a large-scale multi-machine and multi-card training method was proposed, which successfully achieved multi-machine and multi-card parallelism in both homogeneous and heterogeneous mixed models. The BYOL algorithm and the PRP algorithm were trained and evaluated on a large scale using the Slurm framework and the Horovod framework, respectively, which greatly shortened the training time of large models on large data sets.

[0079] (2) To address the problem of non-convergence of training loss under large batch sizes, we used appropriate optimizers such as lars, lambda, and yogi, as well as appropriate parameter adjustment strategies, to solve the problem of model accuracy degradation during large batch size training of ultra-large data sets. We also further expanded the number of machines involved in training and the batch size, improving model accuracy.

[0080] (3) Using IB / RDMA for multi-machine communication and Apex-based mixed-precision acceleration for training further speeds up training and reduces resource consumption;

[0081] (4) A distributed training method based on a hybrid heterogeneous platform of multiple models, including DGX2, DGX1, and AGX, is proposed. This method facilitates the full use of existing heterogeneous devices to carry out multi-machine and multi-card parallel training under resource-constrained conditions. It also provides ideas and references for subsequent larger-scale model training based on hybrid heterogeneous platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0083] Figure 1 It is a flow chart of a large-scale multi-machine multi-card pre-training method provided by an embodiment of the present invention.

[0084] Figure 2 It is a schematic diagram of the multi-queue application mechanism for Cloud Brain I hybrid model training provided by an embodiment of the present invention.

[0085] Figure 3 It is a structural block diagram of a large-scale multi-machine multi-card pre-training system provided by an embodiment of the present invention;

[0086] In the figure: 1. Unsupervised feature learning module; 2. Video semantic unsupervised learning module; 3. Multi-machine and multi-card pre-training module. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0088] In view of the problems existing in the prior art, the present invention provides a large-scale multi-machine and multi-card pre-training method, system, device and server cluster. The present invention is described in detail below with reference to the accompanying drawings.

[0089] like Figure 1 As shown, the large-scale multi-machine multi-card pre-training method provided by the embodiment of the present invention includes the following steps:

[0090] S101, unsupervised feature learning: using BYOL algorithm for multi-machine and multi-card deployment;

[0091] S102, video semantic unsupervised learning: using the PRP algorithm for multi-machine and multi-card deployment;

[0092] S103, multi-machine and multi-card pre-training: perform Cloud Brain I mixed model training.

[0093] The schematic diagram of the multi-queue application mechanism for hybrid model training of Cloud Brain I provided by the embodiment of the present invention is as follows Figure 2 shown.

[0094] like Figure 3 As shown, the large-scale multi-machine multi-card pre-training system provided by the embodiment of the present invention includes:

[0095] Unsupervised feature learning module 1, used for multi-machine and multi-card deployment using BYOL algorithm;

[0096] Video semantic unsupervised learning module 2, used for multi-machine and multi-card deployment using the PRP algorithm;

[0097] The multi-machine and multi-card pre-training module 3 is used for Cloud Brain I hybrid model training.

[0098] The technical solution of the present invention is further described below in conjunction with specific embodiments.

[0099] Example

[0100] 1. Use Pengcheng Cloud Brain I to perform multi-machine and multi-card (GPU) parallel training;

[0101] In response to the need to build an open source, open and shared innovation platform for AI, Pengcheng Laboratory launched the Pengcheng Cloud Brain Phase I platform. Pengcheng Cloud Brain I is a large cluster system built with NVIDIA GPU servers as the infrastructure. As an AI large scientific device to support the construction of a better AI ecosystem, Pengcheng Cloud Brain I has cluster management tools and resource scheduling platforms, and supports running AI tasks in GPU clusters. In the process of building and upgrading smart cities, the amount of data has increased dramatically, and as artificial intelligence tasks become more complex and diverse, the scale of models is also increasing. At present, there are many needs to use large-scale data to train large models in practical applications. Using multiple machines and multiple cards to carry out distributed training is a necessary way to meet such needs. Therefore, large-scale multi-machine and multi-card distributed training based on Pengcheng Cloud Brain I can fully tap its cluster advantages and parallelization capabilities, significantly improve model training efficiency, improve model training effects, and give full play to the role of large scientific devices.

[0102] The large-scale multi-machine and multi-card training task based on Pengcheng Cloud Brain I has two main goals: the first is unsupervised feature learning, using the BYOL algorithm to train the Imagenet2012 dataset (about 1.2 million images), reducing the training time from 7 days to 0.25 days; the second is video semantic unsupervised learning, using the PRP algorithm to train the Kinetics 400 dataset (about 50 million frames of short videos), reducing the training time from 10 days to 0.5 days.

[0103] The large-scale multi-machine and multi-card pre-training method provided by the present invention uses 25 DGX2s with a total of 400 V100s to complete the test, with a maximum batch size of 76,800. Currently, few units at home and abroad can complete large-scale multi-machine and multi-card operations with such powerful computing power. The multi-machine and multi-card large-scale training experiment involved in the present invention has a high batch size and a short training time compression, which not only verifies the parallel capability of the Pengcheng Cloud Brain I large scientific device, but also further expands the cluster scale of parallel training. It has great guiding significance for the feasibility and specific implementation methods of distributed training using ultra-large-scale clusters. At the same time, the model accuracy also shows good performance in the downstream task evaluation after pre-training. The heterogeneous hybrid machine model training scheme involved provides an implementation idea for large-scale model training based on hybrid heterogeneous platforms.

[0104] The implementation of the unsupervised feature learning application example provided by the present invention is as follows:

[0105] (1) The benchmarking work is to use resnet50 as the base network and a total batch size of 4096 to train for 200 rounds to obtain a pre-trained model, and then evaluate the ImageNetLinearClassification task based on the pre-trained model. In the evaluation task, a total of 100 rounds of training are performed. The author uses a single machine with 8 cards and a total batch size of 256. The highest top1 accuracy obtained on the validation set is 67.10;

[0106] (2) The present invention uses resnet101 as the base network and a total batch size of 12288 for 200 rounds of training to obtain a pre-trained model, and evaluates the ImageNetLinearClassification task based on the pre-trained model. In the evaluation task, a total of 100 rounds of training are performed, and a single machine with 16 cards and a total batch size of 2048 are used for testing. The highest top1 accuracy obtained on the validation set is 69.294;

[0107] (3) Using the same parameters as the author, the highest top1 accuracy obtained by the present invention on the validation set is 67.068, while the highest data published by the author is 67.10, which are basically consistent.

[0108] (4) For the application of unsupervised learning of video semantics, since there is no multi-machine multi-card program for the PRP algorithm, the original single-machine multi-card program was modified to complete multi-machine multi-card training and testing.

[0109] Based on the Pengcheng Cloud Brain I platform, the present invention proposes and implements a large-scale multi-machine multi-card distributed training method, and successfully realizes multi-machine multi-card parallelism of homogeneous and heterogeneous hybrid models. Based on the slurm framework and the Horovod framework, the entire process of large-scale multi-machine multi-card training and evaluation of the BYOL algorithm and the PRP algorithm is implemented, including environment configuration, task configuration, communication configuration, task acceleration, etc., which greatly shortens the training time of large models on large data sets, and also further expands the scale of machines involved in training. The training time compression of the two sets of multi-machine multi-card large-scale computing algorithms reaches the expected goal.

[0110] The completion status of multiple machines and multiple cards of the same model in parallel is as follows:

[0111] (1) We used 25 DGX2s with a total of 400 V100s, deployed the slurm framework, configured a multi-machine parallel environment, combined multiple acceleration strategies, and adopted the unsupervised feature learning BYOL algorithm to train the Imagenet2012 dataset and obtain a pre-trained model. This reduced the training time from 7 days to 5 hours and 6 minutes, achieving the expected goal.

[0112] (2) We used 14 DGX2s with a total of 224 V100s, deployed the Horovod framework, configured a multi-machine parallel environment, combined multiple acceleration strategies, and adopted the video semantic unsupervised learning PRP algorithm to train the Kinetics400 dataset, shortening the training time from 10 days to 11 hours and 42 minutes, achieving the expected goal.

[0113] (3) For mixed models, such as DGX1, DGX2, AGX and other models on Cloud Brain I, mixed training of multiple models of the above two algorithms is implemented.

[0114] 2. The present invention is operated on multiple servers using multiple machines and multiple cards.

[0115] Currently, there are three server models on Pengcheng Cloud Brain I, DGX1, DGX2, and AGX. The specific parameter configurations are shown in Table 1.

[0116] Table 1 Server configuration

[0117] model IB network card power Number of CPU cores Memory V100 DGX2 8 per unit 350W 96 cores 1.5T Single machine 16 cards DGX1 2 per unit 163W 80 cores 0.5T Single machine 8 cards AGX 2 per unit 300W 96 cores 1.5T Single machine 8 cards

[0118] (2.1) Unsupervised feature learning: BYOL algorithm multi-machine multi-card deployment solution

[0119] Using N DGX2s with 16×N V100s, the unsupervised feature learning BYOL algorithm was used to train the Imagenet2012 dataset to obtain a pre-trained model, which reduced the training time from 7 days to 5 hours and 6 minutes. The general plan is: set up a CPU server as the master node, and other GPU servers as computing nodes. The master node can be shared with one of the GPU servers; the master node and each sub-node submit the corresponding deployment script.

[0120] The specific deployment process is as follows:

[0121] 101. Multi-machine deployment based on slurm on Cloud Brain I. Since the control node (master node) does not participate in the calculation, it is only necessary to apply for the CPU for the control node;

[0122] Take the DGX2 machine as an example to illustrate the detailed configuration of node parameters.

[0123] (1) Configure the operating environment for each node in a mirrored manner.

[0124] (2) The configuration of the control node is: the control node task does not apply for GPU, the number of CPU cores applies for 6 cores, the memory applies for 100G, and it is set as the backbone task; the configuration of the sub-node is: each sub-node task applies for 16 GPUs, the number of CPU cores applies for 80 cores, and the memory applies for 1T. Since the control node and the sub-nodes can share a server, the total number of machines used is equal to the number of sub-nodes.

[0125] (3) Both the control node and the sub-nodes use IB / RDMA for multi-machine communication and configure half of the memory as shared memory.

[0126] (4) Configure the startup command and training script to start running parallel training tasks.

[0127] 102.Configure a multi-machine slurm environment based on debug mode.

[0128] Debug the DGX2 machine through debug mode. First, enter the task executed on DGX2 through SSH; then, if the slurm-related software is not installed in the environment, install it. If the slurm-related software has been pre-installed, you can verify it through the slurmd-V and slurmd-C commands, and check the number of CPU cores; finally, in the cloud brain version of slurm multi-machine deployment, you need to modify masterip.txt and slaveip.txt, add the master node (control node) IP in masterip.txt, add the child node (computing node) IP in slaveip.txt, and modify the ControlMachine variable in the slurm_autoconfig.sh script, and then execute the bash slurm_autoconfig.sh script to complete the configuration of the multi-machine slurm environment.

[0129] 103. Start Cloud Brain I high-speed multi-machine communication IB / RDMA

[0130] (1) Nvidia NGC19.10 is selected as the base image. The image link is: https: / / docs.nvidia.com / deeplearning / frameworks / pytorch-release-notes / rel_19-10.html#rel_19-10.

[0131] (2) Specify the IB network card in the training script:

[0132]

[0133] 104. Acceleration solutions for large-scale tasks

[0134] The following measures are taken to speed up the training process.

[0135] (1) Dataset storage acceleration: By opening a dedicated dataset storage space and mounting it to memory, the dataset can be used continuously without restarting, and the purpose of accelerating training can be achieved by accelerating data reading;

[0136] (2) Using IB / RDMA for multi-machine communication, the training speed can be improved by accelerating the data exchange process between multiple machines during training;

[0137] (3) Using Apex mixed precision for training, the video memory usage is reduced by half compared to single-precision floating-point training, so the batch size can also be doubled;

[0138] (4) Use optimizers suitable for large-scale batch sizes, such as lars, lambda, yogi, etc., to increase the training speed through optimizer acceleration;

[0139] (5) The allocation of CPU cores is optimized for slurm. When the total number of cores is constant, the efficiency of CPU allocation to jobs is improved as much as possible.

[0140] 105. Based on the above steps, the multi-machine multi-card parallel version of the BYOL algorithm under the slurm framework is used for large-scale multi-machine multi-card model pre-training. The specific running results are as follows:

[0141] Table 2 BYOL algorithm multi-machine multi-card training results

[0142]

[0143]

[0144] 106. Use a single machine with multiple cards to evaluate the pre-trained model of the BYOL algorithm.

[0145] The author of the BYOL algorithm used resnet50 as the base network and trained for 200 rounds with a total batchsize of 4096 to obtain a pre-trained model. Based on the pre-trained model, a single machine with 8 graphics cards and a total batchsize of 256 was used to evaluate the ImageNet LinearClassification task. In the 100 rounds of training for the evaluation task, the highest top1-accuracy on the validation set was 67.10.

[0146] The pre-trained model used in the evaluation of the present invention is: a pre-trained model obtained by training 200 rounds of the base network resnet101 (with larger parameters than resnet50) using 8 DGX2s with a total of 128 V100s, a total batchsize of 12288, and pre-training model; and based on the pre-trained model, a single machine with 16 cards and a total batchsize of 2048 are used to evaluate the ImageNet Linear Classification task. In the 100 rounds of training for the evaluation task, the highest top1-accuracy on the validation set is 69.294. Compared with the original author, the model performance has been improved to a certain extent.

[0147] Table 3 ImageNet Linear Classification task evaluation results

[0148]

[0149]

[0150] (2.2) PRP algorithm multi-machine multi-card deployment solution

[0151] 201. Deploy the Horovod framework to Pengcheng Cloud Brain I.

[0152] First, install and deploy the software environment required by Horovod on the node through the image, and set up ssh password-free login. When starting the task, select the image of the software required by Horovod that has been installed in Cloud Brain I, and select nccl version 2.7.8.

[0153] Then, add the execution of the ssh login script to the task startup command to complete the multi-machine ssh password-free login. The multi-machine password-free configuration script has the following execution steps: first generate the secret key (including private key and public key), copy the secret key to the shared storage directory of the Cloud Brain platform, and create a new ssh login shell script in the directory. The script content is to distribute the secret key to the .ssh path of the root directory of each node and modify the corresponding permissions. This step is convenient for logging into the machine where each task is located for debugging through the Cloud Brain debug mode later.

[0154] 202. From the deployment of the environment to the start of training, in the intermediate steps, except that the master node can apply for GPU as a computing node at the same time, the node parameter configuration, startup of Cloud Brain I high-speed multi-machine communication IB / RDMA and the acceleration plan for large-scale tasks are the same as above.

[0155] 203. Change the PRP algorithm to a multi-machine multi-card parallel version under the Horovod framework. Based on the above steps, carry out large-scale multi-machine multi-card model pre-training. The specific operation results are as follows:

[0156] Table 4 PRP algorithm multi-machine multi-card training results

[0157] Cluster Configuration Batchsize IB network card Data storage Training time 8 machines 128 cards 36*128=4608 1 wire harddisk 20h 30m 12 machines 192 cards 32*192=6144 1 wire harddisk 14h 7m 13 machines 208 cards 32*208=6656 1 wire Memory 12h 25m 14 machines 224 cards 36*224=8064 1 wire Memory 11h 42m

[0158] (2.3) Cloud Brain I Hybrid Model Training Program

[0159] On the Cloud Brain I platform, a multi-queue application method is used to achieve mixed training of cards of different models. This method needs to solve the problem of modifying / etc / hosts under each queue. Since etc / hosts is actually mounted from an external source file, modifying the source file and remounting it can solve the problem of erc / hosts not being able to mount. The pod_id of the source file is obtained through script writing, and the IP and host name of each queue are improved. After remounting, the / etc / hosts in each queue will have the IP and host name of all queues to ensure communication between machines.

[0160] The specific flow chart is as follows Figure 2 shown.

[0161] Take an example to illustrate the steps of applying for multi-queue resources: use 5 DGX2s + 1 DGX1 + 1 AGX to achieve parallelism. Queue 1, apply for 1 DGX2, start 1 task, set as the master node; Queue 2, apply for 4 DGX2s, start 4 tasks, set as the child node; Queue 3, apply for 1 AGX, start 1 task, set as the child node; Queue 4, apply for 1 DGX1, start 1 task, set as the child node. The configuration parameters of the task and the training acceleration plan are the same as above.

[0162] The key points of the present invention are:

[0163] (1) Based on the Pengcheng Cloud Brain I platform, a large-scale multi-machine and multi-card training method was proposed, which successfully achieved multi-machine and multi-card parallelism in both homogeneous and heterogeneous mixed models. The BYOL algorithm and the PRP algorithm were trained and evaluated on a large scale using the Slurm framework and the Horovod framework, respectively, which greatly shortened the training time of large models on large data sets.

[0164] (2) To address the problem of non-convergence of training loss under large batch sizes, we used appropriate optimizers such as lars, lambda, and yogi, as well as appropriate parameter adjustment strategies, to solve the problem of model accuracy degradation during large batch size training of ultra-large data sets. We also further expanded the number of machines involved in training and the batch size, improving model accuracy.

[0165] (3) Using IB / RDMA for multi-machine communication and Apex-based mixed-precision acceleration for training further speeds up training and reduces resource consumption;

[0166] (4) A distributed training method based on a hybrid heterogeneous platform of multiple models, including DGX2, DGX1, and AGX, is proposed. This method facilitates the full use of existing heterogeneous devices to carry out multi-machine and multi-card parallel training under resource-constrained conditions. It also provides ideas and references for subsequent larger-scale model training based on hybrid heterogeneous platforms.

[0167] The single machine multi-card code link of the prp algorithm: https: / / github.com / yuanyao366 / PRP;

[0168] The paper link of prp algorithm: https: / / arxiv.org / abs / 2006.11476;

[0169] One of the contributions of the present invention is to modify the prp algorithm into a multi-machine and multi-card version and run it on Pengcheng Cloud Brain 1.

[0170] In the description of the present invention, unless otherwise specified, "plurality" means two or more than two; the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "inner", "outer", "front end", "rear end", "head", "tail", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0171] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0172] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A large-scale multi-machine multi-card pre-training method, characterized in that: The large-scale multi-machine multi-card pre-training method comprises the following steps: Step 1: Unsupervised feature learning based on multi-machine and multi-card parallelism in the slurm framework: BYOL algorithm is used for multi-machine and multi-card deployment, further including: (1) Deploy multiple machines based on Slurm on Cloud Brain I. Since the control node, i.e. the master node, does not participate in the calculation, only the CPU needs to be applied for the control node; If the server model is DGX2, the configuration of node parameters includes: Configure the operating environment for each node in a mirrored manner; The configuration of the control node is: the control node task does not apply for GPU, the number of CPU cores applies for 6 cores, the memory applies for 100G, and it is set as the backbone task; the configuration of the sub-nodes is: each sub-node task applies for 16 GPUs, the number of CPU cores applies for 80 cores, and the memory applies for 1T; according to the control node and sub-nodes sharing a server, the total number of machines used is equal to the number of sub-nodes; Both the control node and the sub-nodes use IB / RDMA for multi-machine communication and configure half of the memory as shared memory; Configure the startup command and training script to start running parallel training tasks; (2) Configure a multi-machine slurm environment based on debug mode; Debug the DGX2 machine in debug mode: enter the task executed on DGX2 through SSH; if slurm-related software is not installed in the environment, install it; if slurm-related software has been pre-installed, verify it through slurmd-V and slurmd -C commands, and check the number of CPU cores; in the cloud brain version of slurm multi-machine deployment, modify masterip.txt and slaveip.txt, add the master node, that is, the IP of the control node, to masterip.txt; add the child node, that is, the IP of the computing node, to slaveip.txt, and modify the ControlMachine variable in the slurm_autoconfig.sh script, and then execute the bash slurm_autoconfig.sh script to complete the configuration of the multi-machine slurm environment; (3) Start Cloud Brain I high-speed multi-machine communication IB / RDMA Choose Nvidia NGC19.10 as the base image; Specify the IB network card in the training script: ①os.environ['NCCL_IB_HCA'] = "mlx5_0"; ②os.environ['NCCL_DEBUG'] = "INFO"; (4) Acceleration solutions for large-scale tasks The following measures are taken to speed up the training process: Dataset storage acceleration: A dedicated dataset storage space is created and mounted on the memory. The dataset is used continuously without restarting the system. Training is accelerated by accelerating data reading. IB / RDMA is used for multi-machine communication to speed up the data interaction between multiple machines in training, thereby improving the training speed. Apex mixed precision is used for training, which reduces the video memory usage by half compared to single-precision floating-point training, so the batch size is doubled. Use an optimizer suitable for large-scale batch sizes, and use the optimizer to accelerate the training speed. Optimized the allocation of CPU cores for slurm, and maximized the efficiency of CPU allocation jobs when the total number of cores is fixed; (5) Based on steps (1) to (4), the multi-machine and multi-card parallel version of the BYOL algorithm under the slurm framework is used for large-scale multi-machine and multi-card model pre-training; (6) Evaluate the pre-trained model of BYOL algorithm using multiple cards on a single machine; Step 2: Multi-machine and multi-card parallel video semantic unsupervised learning based on the Horovod framework: The PRP algorithm is used to deploy multiple machines and multiple cards. The video semantic unsupervised learning includes: (1) Deploy the Horovod framework to Pengcheng Cloud Brain I; Complete the installation and deployment of the software environment required by Horovod on the node through the image, and set up ssh password-free login; when starting the task, select the image of the software required by Horovod that has been installed in Cloud Brain I; add the execution of the ssh login script to the task startup command to complete the multi-machine ssh password-free login; (2) From the deployment of the environment to the start of training, the intermediate steps are the same as above, except that the master node can apply for a GPU as a computing node at the same time. The node parameter configuration, start of Cloud Brain I high-speed multi-machine communication IB / RDMA and the acceleration solution for large-scale tasks are the same as above; (3) The PRP algorithm is changed to a multi-machine and multi-card parallel version under the Horovod framework. Based on steps (1) to (2), large-scale multi-machine and multi-card model pre-training is carried out. Step three: Conduct multi-machine, multi-card mixed model training.

2. The large-scale multi-machine multi-card pre-training method as claimed in claim 1, characterized in that: The step 1 further comprises: We used N DGX2s with 16×N V100s to train the Imagenet2012 dataset using the unsupervised feature learning BYOL algorithm to obtain a pre-trained model, reducing the training time from 7 days to 5 hours and 6 minutes. Set a CPU server as the master node and other GPU servers as computing nodes. The master node is shared with one of the GPU servers. Submit the corresponding deployment scripts to the master node and each sub-node.

3. The large-scale multi-machine multi-card pre-training method as claimed in claim 1, characterized in that: In step (1), the multi-machine password-free configuration script includes: First generate the secret key, including the private key and the public key, copy the secret key to the shared storage directory of the Cloud Brain platform, and create a shell script for ssh login in this directory. The script content is to distribute the secret key to the .ssh path of the root directory of each node and modify the corresponding permissions; this step is convenient for later debugging through the Cloud Brain debug mode, without password login to the machine where each task is located.

4. The large-scale multi-machine multi-card pre-training method as claimed in claim 2, characterized in that: In step 3, the multi-machine multi-card pre-training includes: On the Cloud Brain I platform, a multi-queue application method is used to implement mixed training of cards of different models. The source file is modified and remounted. The pod_id of the source file is obtained through scripting, and the IP and host name of each queue are improved.

5. A large-scale multi-machine multi-card pre-training system using the large-scale multi-machine multi-card pre-training method according to any one of claims 1 to 4, characterized in that: The large-scale multi-machine multi-card pre-training system comprises: Unsupervised feature learning module for multi-machine and multi-card deployment using BYOL algorithm; Video semantic unsupervised learning module, used for multi-machine and multi-card deployment using the PRP algorithm; The multi-machine and multi-card pre-training module is used for multi-machine and multi-card mixed model training.

6. A computer-readable storage medium storing a computer program of the large-scale multi-machine multi-card pre-training method according to any one of claims 1 to 4, wherein when the computer program is executed by a processor, the processor executes the following steps: (1) Unsupervised feature learning: Use the BYOL algorithm for multi-machine and multi-card deployment; (2) Unsupervised learning of video semantics: using the PRP algorithm for multi-machine and multi-card deployment; (3) Multi-machine and multi-card pre-training: Perform multi-machine and multi-card mixed model training.