Multi-node GPU cluster scheduling method applied to deep learning training scene

Abstract

The invention discloses a multi-node GPU cluster scheduling method applied to a deep learning training scene. The method mainly comprises the following steps: S1, building a deep learning model; s2, inputting training data and segmenting the training data according to the number of GPUs; and S3, carrying out fragmentation on the model parameters. And S4, carrying out forward propagation on the model and carrying out data communication between GPUs. And S5, carrying out back propagation on the model and carrying out data communication between the GPUs. And S6, continuously training and iterating, stopping training after the maximum number of training iterations is reached, and storing the model weight. According to the method, multi-node GPU cluster scheduling is optimized, the communication mode is improved, the utilization rate of the video in a deep learning scene is remarkably improved, and training of a larger model becomes possible under the condition that the hardware cost is not increased.

Description

Technical Field

[0001] The present invention relates to the fields of computer technology, high performance computing, and artificial intelligence, and in particular to the field of deep learning, and specifically to a multi-node GPU cluster scheduling method applied to deep learning training scenarios. Background Art

[0002] In recent years, the field of artificial intelligence has developed rapidly. In particular, technologies based on deep learning have achieved great results, which has greatly promoted scientific and technological progress and social development. However, the increasingly large scale of data and the complex network scale have posed great challenges to the training of deep learning models. Multi-node GPU calls have always attracted much attention in the field of deep learning, so distributed training methods have been derived. Taking the training of large language models as an example, even if some distributed training methods are applied, the large number of parameters still makes the GPU memory size a bottleneck. Therefore, how to effectively reduce the GPU memory usage in distributed training and improve the utilization of memory has become a key research direction. Summary of the invention

[0003] The purpose of the present invention is to provide a multi-node GPU cluster scheduling method for deep learning training scenarios, which can effectively reduce the memory usage during model training and improve the utilization of memory. To achieve the above purpose, the present invention provides the following technical solution, which includes the following steps:

[0004] Step S1: Build a deep learning neural network.

[0005] Step S2: Input the training data with Batch Size N into the neural network model, and then perform data segmentation.

[0006] Furthermore, if the number of GPUs in the GPU cluster is n, each GPU obtains N / n data.

[0007] Step S3: Sharding the model parameters. In order to reduce the memory usage, the model weight parameters, model gradient parameters and optimizer state parameters are sharded, that is, only 1 / n of the original model parameters are stored on each GPU.

[0008] Step S4: After completing the model parameter sharding, perform forward propagation to obtain the loss.

[0009] Furthermore, forward propagation mainly includes some activation value calculations, intermediate feature calculations, and loss calculations, and does not involve the update of model parameters. Activation values ​​and intermediate features are directly optimized using FP16. Specifically, if data operations overflow, they are saved using FP32, otherwise they are saved using FP16. After each GPU completes its own forward propagation, the data is communicated between GPUs. The data here mainly includes: the activation value, intermediate features, and loss of the model.

[0010] Furthermore, the communication between GPUs is as follows:

[0011] Step S4-1: n GPUs are formed into a communication ring structure, and the data on each GPU is evenly divided into n parts;

[0012] Step S4-2: For the jth (j∈{0,1,2…n}) GPU, this GPU will send the jth data to the j+1th GPU and receive the j-1th data from the j-1th GPU. Let the intermediate variable K j =j-1. In particular, since the communication structure is a ring, when j-1=0, it is considered that j-1=n; when j+1>n, it is considered that j+1=1.

[0013] Step S4-3: Then, the jth GPU will combine the received data with its Kth j The data is integrated and then sent to the j+1th GPU. Update K j , let K j =K j -1. Wait for all GPUs in the cluster to complete communication, that is, all GPUs have sent their own data and received new data.

[0014] Step S4-4: Repeat step S4-3 for n cumulative execution times. At this point, each GPU contains a copy of the final data.

[0015] Step S4-5: After completing step S4-4, starting from the first GPU, the data is sent to the next GPU in sequence. When the next GPU receives the data, it integrates it with its own data and sends it to the GPU behind it. When the data is passed to the last GPU (the nth), all the required data is collected on this GPU.

[0016] Step S4-6: The nth GPU broadcasts and directly replaces the data on the remaining GPUs.

[0017] Step S5: After completing the forward propagation, perform back propagation based on the loss and update the model parameters, which involves model gradient parameter calculation, optimizer parameter update, and model weight parameter update.

[0018] Furthermore, after each GPU completes its own back propagation, it uses the GPU communication method in step S4 to communicate data. The data here includes: model gradient parameters, optimizer parameters and model weight parameters.

[0019] Furthermore, during the back-propagation process, the model parameters involve a dynamic storage method. The storage types of model parameters are divided into FP32 (saved using 32-bit floating point numbers), BF16 (saved using 16-bit floating point numbers), and FP16 (saved using 16-bit floating point numbers). FP32 contains 8 exponent bits, 23 mantissa bits, and 1 sign bit. BF16 contains 8 exponent bits, 7 mantissa bits, and 1 sign bit. FP16 contains 5 exponent bits, 10 mantissa bits, and 1 sign bit.

[0020] Their values ​​are expressed by the following formula:

[0021] V FP32 =(-1) sign ×2 exponent-127 ×(1.0+fraction)

[0022] V FP16 =(-1) sign ×2 exponent-15 ×(1.0+fraction)

[0023] V BF16 (-1) sign ×2 exponent-127 ×(1.0+fraction)

[0024] Where sign represents the sign bit, and its value is 0 or 1. exponent represents the decimal integer value corresponding to the exponent bit, and its value range is [0, 2 l ], l is the exponent bit length. The fractional part is represented by the binary representation of the mantissa bit length, and fraction is the decimal value corresponding to the binary. FP32 、V FP16 、V BF16 Corresponding to different parameter value types.

[0025] Furthermore, the standard for determining the parameter saving type is whether the parameter overflows (exceeds the maximum representation range of FP16 or) when applying FP16 to a certain operation. If the value does not overflow, FP16 is selected to save the value, otherwise BF16 is used to save it.

[0026] Step S6: When the number of training iterations reaches the set maximum value, the training ends and the model weights are saved, otherwise steps S2 to S5 are repeated.

[0027] Beneficial effects that can be achieved by applying the above steps:

[0028] (1) The present invention divides the data into different GPUs, reduces the pressure of data on the video memory, and increases the throughput of network training.

[0029] (2) The present invention divides the model parameters, each GPU contains part of the model parameters, and the entire network contains only one copy of the model parameters, which reduces data redundancy and further improves the utilization rate of the video memory. Under the same hardware conditions, a larger model can be trained.

[0030] (3) The present invention ensures training accuracy and takes training speed into consideration by setting the data saving type, and reduces the occupancy of video memory.

[0031] (4) The communication method between GPUs set up in the present invention aggregates all data, reduces the total number of communications and communication volume, and has linear acceleration capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the present invention;

[0033] Figure 2 A schematic diagram of the inter-GPU communication method of the present invention;

[0034] Figure 3 A schematic diagram of video memory occupation division of the present invention; DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. The following table explains some terms used in the present invention.

[0036] English logo Chinese explanation English logo Chinese explanation FP16 (floatpoint16) 16-bit floating point FP32 (floatpoint32) 32-bit floating point BF16 (brainfloat16) 16-bit floating point BatchSize The batch size of the data momentum momentum variance variance

[0037] The present invention relates to a multi-node GPU cluster scheduling method applied to a deep learning training scenario, which specifically comprises the following steps:

[0038] Step S1: Build a deep learning neural network.

[0039] Step S2: Input the training data with batch size N into the neural network model, and then split the data. Assume that the number of GPUs in the GPU cluster is n, and each GPU obtains N / n data (it is necessary to ensure that N / n is a positive integer).

[0040] like Figure 2As shown, during the training process, the actual video memory occupied is divided into three categories: deep learning framework occupation, model parameter related occupation, and feature related occupation. The video memory occupied by the framework is related to the deep learning framework actually applied. The video memory occupied by features is related to the specific structure and calculation process of the model, and contains some intermediate results and activation values, which will also cause video memory fragmentation and make it difficult to optimize. Therefore, in order to implement a general video memory optimization solution, the present invention mainly optimizes the video memory occupied by model parameters, which is usually the largest part of the video memory. The video memory occupied by model parameters can be specifically divided into model weight parameter occupation, model gradient parameter occupation, and optimizer state parameter occupation.

[0041] Step S3: Shard the model parameters. In traditional distributed training methods, model parameters (model weight parameters, model gradient parameters, and optimizer state parameters) are copied between GPUs, resulting in redundancy of video memory and low utilization. In order to reduce video memory usage, the model weight parameters, model gradient parameters, and optimizer state parameters are sharded, that is, only 1 / n of the original model parameters are stored on each GPU. Model weights are stored in FP16 or BF16, model gradients are stored in FP16 or BF16, and optimizer state parameters are still stored in FP32.

[0042] Example 1

[0043] Assume that the parameter amount of the model weight (FP16 or BF16 storage) is δ, corresponding to the storage space required of 2δ bytes. The parameter amount of the model gradient (FP16 or BF16 storage) is still δ, and also requires 2δ bytes of storage space. Assuming that the ADAM optimizer is used, the parameter amount of the optimizer state (FP32 storage) includes the model parameter backup of FP32 (requiring 4δ bytes of storage), the momentum of FP32 (requiring 4δ bytes of storage) and the variance of FP32 (requiring 4δ bytes of storage), requiring a total of 12δ bytes of storage space. That is, each GPU needs to occupy approximately 2δ+2δ+(4δ+4δ+4δ)=16δ bytes of video memory to save the model parameters. Only the occupation of video memory related to the model parameters is considered here. The actual occupied video memory will be higher than 16δ bytes. The present invention performs slicing processing on the model parameters, that is, each GPU only stores 1 / n parameters. Therefore, the video memory occupied by the model parameters on each card is: 16δ / n.

[0044] Step S4: After completing the model parameter slicing, forward propagation is performed to obtain the loss. Forward propagation mainly includes some activation value calculations, intermediate feature calculations, and loss calculations, and does not involve the update of model parameters. The activation value and intermediate features are directly optimized using FP16. Specifically: if the data operation overflows, it is saved using FP32, otherwise it is saved using FP16. After each GPU completes its own forward propagation, the data is communicated between GPUs. The data here mainly includes: the activation value of the model, the intermediate features, and the loss. The following is a specific solution for communication between GPUs.

[0045] Embodiment 2:

[0046] like Figure 3 , the communication between GPUs is as follows:

[0047] Step S4-1: n GPUs are formed into a communication ring structure, and the data on each GPU is evenly divided into n parts;

[0048] Step S4-2: For the jth (j∈{0,1,2…n}) GPU, this GPU will send the jth data to the j+1th GPU and receive the j-1th data from the j-1th GPU. Let the intermediate variable K j =j-1. In particular, since the communication structure is a ring, when j-1=0, it is considered that j-1=n; when j+1>n, it is considered that j+1=1.

[0049] Step S4-3: Then, the jth GPU will combine the received data with its Kth j The data is integrated and then sent to the j+1th GPU. Update K j , let K j =K j -1. Wait for all GPUs in the cluster to complete communication, that is, all GPUs have sent their own data and received new data.

[0050] Step S4-4: Repeat step S4-3 for n cumulative execution times. At this point, each GPU contains a copy of the final data.

[0051] Step S4-5: After completing step S4-4, starting from the first GPU, the data is sent to the next GPU in sequence. When the next GPU receives the data, it integrates it with its own data and sends it to the GPU behind it. When the data is passed to the last GPU (the nth), all the required data is collected on this GPU.

[0052] Step S4-6: The nth GPU broadcasts and directly replaces the data on the remaining GPUs.

[0053] Step S5: After completing the forward propagation, perform back propagation based on the loss and update the model parameters. This involves model gradient parameter calculation, optimizer parameter update, and model weight parameter update. After each GPU completes its own back propagation, it uses the GPU communication method in step S4 above to communicate data. The data here includes: model gradient parameters, optimizer parameters, and model weight parameters.

[0054] During the back-propagation process, the model parameters involve a dynamic storage method. The storage types of model parameters are divided into FP32 (saved using 32-bit floating point numbers), BF16 (saved using 16-bit floating point numbers), and FP16 (saved using 16-bit floating point numbers). FP32 contains 8 exponent bits, 23 mantissa bits, and 1 sign bit. BF16 contains 8 exponent bits, 7 mantissa bits, and 1 sign bit. FP16 contains 5 exponent bits, 10 mantissa bits, and 1 sign bit.

[0055] Their values ​​are expressed by the following formula:

[0056] V FP32 =(-1) sign ×2 exponent-127 ×(1.0+fraction)

[0057] V FP16 =(-1) sign ×2 exponent-15 ×(1.0+fraction)

[0058] V BF16 (-1) sign ×2 exponent-127 ×(1.0+fraction)

[0059] Where sign represents the sign bit, and its value is 0 or 1. exponent represents the decimal integer value corresponding to the exponent bit, and its value range is [0, 2 l ], l is the exponent bit length. The fractional part is represented by the binary representation of the mantissa bit length, and fraction is the decimal value corresponding to the binary. FP32 、V FP16 、V BF16 Corresponding to different parameter value types.

[0060] Compared with traditional FP16, BF16 can represent the same integer values ​​as FP32, with a wider range and less prone to overflow in calculations, but the range of decimals that can be represented is narrower and the accuracy is relatively poor. In order to ensure high training accuracy, the optimizer state parameters are saved in FP32, and the remaining parameters are saved in FP16 or BF16. The standard for determining the parameter saving type is: whether the parameter overflows (exceeds the maximum representation range of FP16) when FP16 is used for a certain operation. If no value overflow occurs, choose to use FP16 to save the value, otherwise use BF16 to save it.

[0061] Step S6: When the number of training iterations reaches the set maximum value, the training is terminated and the model weights are saved, otherwise the training iteration process continues (repeating steps S2 to S5).

Claims

1. A multi-node GPU cluster scheduling method applied to deep learning training scenarios, characterized by: The following steps are involved: Step S1: Build a deep learning neural network. Step S2: Input the training data with batch size N into the neural network model, and then split the data. For a multi-node GPU cluster with n GPUs, each GPU obtains N / n data. Step S3: Sharding the model parameters. In order to reduce the memory usage, the model weight parameters, model gradient parameters and optimizer state parameters are sharded, that is, only 1 / n of the original model parameters are stored on each GPU. Step S4: After completing the model parameter sharding, forward propagation is performed to obtain the loss. Forward propagation mainly includes some activation value calculations, intermediate feature calculations, and loss calculations, and does not involve the update of model parameters. The activation value and intermediate features are directly optimized using FP16. After each GPU completes its own forward propagation, the data is communicated between GPUs. The data here mainly includes: the activation value, intermediate features, and loss of the model. Step S5: After completing the forward propagation, perform back propagation based on the loss and update the model parameters. This involves model gradient parameter calculation, optimizer parameter update, and model weight parameter update. After each GPU completes its own back propagation, it uses the GPU communication method in step S4 above to communicate data. The data here includes: model gradient parameters, optimizer parameters, and model weight parameters. Step S6: When the number of training iterations reaches the set maximum value, the training ends and the model weights are saved, otherwise steps S2 to S5 are repeated.

2. According to a multi-node GPU cluster scheduling method applied to deep learning training scenarios according to claim 1, it is characterized in that: The deep learning neural network in step S1 may have a variety of structures, including but not limited to neural networks built with deep learning frameworks such as pytorch and tensorflow.

3. The multi-node GPU cluster scheduling method for deep learning training scenarios according to claim 1, characterized in that: The step S2 divides the training data. For a multi-node GPU cluster with n GPUs, each GPU obtains N / n data.

4. The multi-node GPU cluster scheduling method for deep learning training scenarios according to claim 1, characterized in that: In order to reduce the memory usage, step S3 applies a model sharding method, which shards the model weight parameters, model gradient parameters and optimizer state parameters, that is, only 1 / n of the original model parameters are stored on each GPU.

5. The multi-node GPU cluster scheduling method for deep learning training scenarios according to claim 1, characterized in that: The model weights in step S3 are stored using FP16 or BF16, the model gradients are stored using FP16 or BF16, and the optimizer state parameters are still stored using FP32.

6. The multi-node GPU cluster scheduling method for deep learning training scenarios according to claim 1, characterized in that: The activation value in step S4, the intermediate feature is directly optimized using FP16, and the rule is: if the data operation overflows, it is saved using FP32, otherwise it is saved using FP16.

7. The multi-node GPU cluster scheduling method for deep learning training scenarios according to claim 1, characterized in that: The communication method between GPUs in step S4 is as follows: Step S4-1: n GPUs are formed into a communication ring structure, and the data on each GPU is evenly divided into n parts; Step S4-2: For the jth (j∈{0,1,2…n}) GPU, this GPU will send the jth data to the j+1th GPU and receive the j-1th data from the j-1th GPU. Let the intermediate variable K j =j-1. In particular, since the communication structure is a ring, when j-1=0, it is considered that j-1=n; when j+1>n, it is considered that j+1=1. Step S4-3: Then, the jth GPU will combine the received data with its Kth j The data is integrated and then sent to the j+1th GPU. Update K j , let K j =K j -1. Wait for all GPUs in the cluster to complete communication, that is, all GPUs have sent their own data and received new data. Step S4-4: Repeat step S4-3 for n cumulative execution times. At this point, each GPU contains a copy of the final data. Step S4-5: After completing step S4-4, starting from the first GPU, the data is sent to the next GPU in sequence. When the next GPU receives the data, it integrates it with its own data and sends it to the GPU behind it. When the data is passed to the last GPU (the nth), all the required data is collected on this GPU. Step S4-6: The nth GPU broadcasts and directly replaces the data on the remaining GPUs.

8. The multi-node GPU cluster scheduling method for deep learning training scenarios according to claim 1, characterized in that: The model parameters in step S5 involve a dynamic saving method. Specifically, the saving type of the model parameters is divided into FP32 (saved using 32-bit floating point numbers), BF16 (saved using 16-bit floating point numbers) and FP16 (saved using 16-bit floating point numbers). FP32 contains 8 exponent bits, 23 mantissa bits and 1 sign bit. BF16 contains 8 exponent bits, 7 mantissa bits and 1 sign bit. FP16 contains 5 exponent bits, 10 mantissa bits and 1 sign bit. Their values ​​are expressed by the following formula: V FP32 =(-1) sign ×2 exponent-127 ×(1.0+fraction) V FP16 =(-1) sign ×2 exponent-15 ×(1.0+fraction) V BF16 (-1) sign ×2 exponent-127 ×(1.0+fraction) Where sign represents the sign bit, and its value is 0 or 1. exponent represents the decimal integer value corresponding to the exponent bit, and its value range is [0, 2 l ], l is the exponent bit length. The fractional part is represented by the binary representation of the mantissa bit length, and fraction is the decimal value corresponding to the binary. FP32 、V FP16 、V BF16 Corresponding to different parameter value types. The standard for judging the parameter saving type is: whether the value overflows (exceeds the maximum representation range of FP16) when the parameter is applied to FP16 for a certain operation. If the value does not overflow, choose to use FP16 to save the value, otherwise use BF16 to save.

Images