Method, device and system for controlling global batch size, medium and equipment
By dynamically adjusting the training hyperparameters during deep learning training, the model instability caused by GBS fluctuations is resolved, achieving higher training accuracy and stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-21
- Publication Date
- 2026-04-07
AI Technical Summary
In large-scale distributed deep learning training, fluctuations in Global Batch Size (GBS) can cause model training instability or failure. Existing technologies mainly mitigate the impact through remedial measures but have failed to effectively suppress the fluctuations.
Before each training step begins, the total parallel scale of the current training data is dynamically calculated. Based on the expected GBS and the total parallel scale, the training hyperparameters are dynamically adjusted to ensure that the GBS value of each training step is consistent with the expected GBS value. This includes obtaining the expected GBS, calculating the total parallel scale of the data, and adjusting training hyperparameters such as the number of micro-batches and the number of gradient accumulations.
It reduces problems such as gradient noise, gradient failure, and learning rate failure caused by GBS fluctuations, improves the accuracy and stability of model training, and reduces the probability of training failure.
Smart Images
Figure CN121808368A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of deep learning technology, and in particular to a method, apparatus, system, medium, and device for controlling global batch size. Background Technology
[0002] As deep learning models grow larger, the requirements for the stability of model training also increase.
[0003] Typically, large-scale model training employs large-scale distributed training. In large-scale clusters, the probability of temporary failures of single nodes or network links increases linearly with the number of nodes. These failures can directly lead to training interruptions or data transmission errors. To address this instability, the distributed scale is dynamically adjusted to maintain training continuity and preserve final accuracy. However, this process alters the number of nodes, causing fluctuations in the Global Batch Size (GBS). Since hyperparameters such as learning rate and regularization are usually strongly correlated with batch size, GBS fluctuations can lead to a decrease in model convergence speed or even divergence, resulting in unstable or failed model training.
[0004] Currently, in order to solve the problem of GBS fluctuations, the main approach in actual training is to adjust and process the GBS fluctuations after they occur, i.e., to take remedial measures after the GBS fluctuations occur. This delays the impact of the GBS fluctuations, but does not solve the problem of GBS fluctuations. Therefore, there is an urgent need for a solution that can suppress GBS fluctuations. Summary of the Invention
[0005] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, system, medium and device for controlling global batch size, which can suppress GBS fluctuations and maintain GBS stability during the distributed training process of deep learning.
[0006] Firstly, a method for controlling the global batch size is provided. This method includes: obtaining the expected GBS during the target training process, where the target training is elastic training or fault-tolerant training in distributed deep learning training, and there are dynamic additions or removals of devices during the target training process; dynamically calculating the total parallel scale of the data currently participating in the training before each training step begins; and dynamically adjusting the training hyperparameters based on the expected GBS and the total parallel scale of the data to ensure that the GBS value of each training step is consistent with the expected GBS value.
[0007] Optionally, before each training step begins, the number of currently active replica groups is obtained, where a replica group indicates a set of devices that jointly maintain a replica of a model; based on the number of replica groups and the data parallelism of a single replica, the total data parallelism currently participating in training is determined.
[0008] Optionally, the training hyperparameters can be dynamically adjusted based on the expected GBS, the total parallel scale of the data, and the model splitting for the target training.
[0009] Optionally, a model splitting index is determined for the target training, which indicates whether pipeline parallelism exists during the training process; the training hyperparameters for the target training are determined based on the model splitting index; wherein, different model splitting indices result in different training hyperparameters.
[0010] Optionally, if the model splitting exponent is greater than 1, the training hyperparameter to be adjusted for the target training is determined as the number of micro-batches, and the adjusted number of micro-batches is determined based on the expected GBS, the micro-batch size and the total parallel scale of the data.
[0011] Optionally, when the model splitting exponent is equal to 1, the training hyperparameter to be adjusted for the target training is the number of gradient accumulations. Based on the expected GBS, micro-batch size and total data parallel scale, the adjusted number of gradient accumulations for the target training is determined.
[0012] Optionally, the number of micro-batches in the current training step is smaller than that in the previous training step, i.e., the pipeline length of the current training step is shortened; or the number of micro-batches in the current training step is larger than that in the previous training step, i.e., the pipeline length of the current training step is increased.
[0013] Optionally, the number of gradient accumulations in the current training step is smaller than that in the previous training step, that is, the number of forward-backward loops in the current training step is reduced; or the number of gradient accumulations in the current training step is larger than that in the previous training step, that is, the number of forward-backward loops in the current training step is increased.
[0014] Optionally, before each training step, the expected GBS for the current training step is determined based on the GBS increment method specified by the target strategy used during training, the total target GBS, and the expected GBS of the previous training step.
[0015] Optionally, the target strategy is a ramp-up strategy, which indicates that the GBS is gradually increased from the first GBS to the second GBS over N training steps; the GBS increment for each training step is determined based on the first GBS, the second GBS, and the total number of training steps; and the expected GBS for the current training step is determined based on the GBS increment and the number of steps in the current training step.
[0016] Optionally, if at the start of training, the number of replica groups participating in training is the number of the first replica groups, and the data parallelism scale of each replica group is the first data parallelism scale, then based on the number of the first replica groups and the first data parallelism scale, the dynamic total data parallelism scale of the first training step is determined, and based on the first GBS, the micro-batch size, and the dynamic total data parallelism scale of the first training step, the number of micro-batches within the first training step period is determined; if before the start of training in the N1th training step, the number of activated replica groups becomes the number of the second replica groups, then based on the number of the second replica groups and the first data parallelism scale, the total data parallelism scale of the N1th training step is determined, and based on the expected GBS, the micro-batch size, and the total data parallelism scale of the N1th training step, the number of micro-batches within the N1th training step period is determined.
[0017] Secondly, a device for controlling the global batch size is provided, comprising: a GBS acquisition unit, a data scale acquisition unit, and a hyperparameter adjustment unit; the GBS acquisition unit is used to acquire the expected GBS during the target training process, where the target training is elastic training or fault-tolerant training in distributed training, and devices dynamically join or leave during the target training process; the data scale acquisition unit is used to dynamically calculate the total parallel scale of the currently participating data before the start of each training step; the hyperparameter adjustment unit is used to dynamically adjust the training hyperparameters based on the expected GBS and the total parallel scale of the data, so that the GBS value of each training step is consistent with the expected GBS value.
[0018] Optionally, the data scale acquisition unit is specifically used to: obtain the number of currently active replica groups before the start of each training step; a replica group indicates a set of devices that jointly maintain a replica of a model; and determine the total data parallelism scale currently participating in training based on the number of replica groups and the data parallelism scale of a single replica.
[0019] Optionally, a hyperparameter tuning unit is used to dynamically adjust training hyperparameters based on the expected GBS, total data parallelism, and model splitting for target training.
[0020] Optionally, the hyperparameter tuning unit is specifically used to determine the model split index for the target training. The model split index is used to indicate whether pipeline parallelism exists during the training process. Based on the model split index, the training hyperparameters to be adjusted for the target training are determined. Different model split indices result in different training hyperparameters.
[0021] Optionally, the hyperparameter tuning unit is specifically used to: determine the number of micro-batches as the training hyperparameter to be tuned for the target training when the model splitting exponent is greater than 1, and determine the adjusted number of micro-batches based on the expected GBS, micro-batch size and total data parallel scale.
[0022] Optionally, the hyperparameter tuning unit is specifically used to: determine the number of gradient accumulations as the training hyperparameter to be tuned for the target training when the model splitting exponent is equal to 1, and determine the adjusted number of gradient accumulations for the target training based on the expected GBS, micro-batch size and total data parallel scale.
[0023] Optionally, the number of micro-batches in the current training step is smaller than that in the previous training step, i.e., the pipeline length of the current training step is shortened; or the number of micro-batches in the current training step is larger than that in the previous training step, i.e., the pipeline length of the previous training step is increased.
[0024] Optionally, the number of gradient accumulations in the current training step is smaller than that in the previous training step, that is, the number of forward-backward loops in the current training step is reduced; or the number of gradient accumulations in the current training step is larger than that in the previous training step, that is, the number of forward-backward loops in the current training step is increased.
[0025] Optionally, the GBS acquisition unit is specifically used to: before each training step, determine the expected GBS for the current training step based on the GBS increment method specified by the target strategy adopted for training, the total target GBS, and the expected GBS of the previous training step.
[0026] Optionally, the target strategy is a ramp-up strategy, which indicates that the GBS is gradually increased from the first GBS to the second GBS over N training steps; the GBS acquisition unit is specifically used to: determine the GBS increment for each training step based on the first GBS, the second GBS, and the total number of training steps, and determine the expected GBS for the current training step based on the GBS increment and the number of steps in the current training step.
[0027] Optionally, the hyperparameter tuning unit is specifically used to: if at the start of training, the number of replica groups participating in training is the number of the first replica groups, and the data parallelism scale of each replica group is the first data parallelism scale, then based on the number of the first replica groups and the first data parallelism scale, determine the dynamic total data parallelism scale of the first training step, and based on the first GBS, the micro-batch size, and the dynamic total data parallelism scale of the first training step, determine the number of micro-batches in the first training step cycle; if before the start of training in the N1th training step, the number of activated replica groups becomes the number of the second replica groups, then based on the number of the second replica groups and the first data parallelism scale, determine the total data parallelism scale of the N1th training step, and based on the expected GBS, the micro-batch size, and the total data parallelism scale of the N1th training step, determine the number of micro-batches in the N1st training step cycle.
[0028] Thirdly, a large-scale distributed deep learning system is provided, comprising: a control plane device and one or more training devices participating in target training, and the training devices including means for controlling the global batch size as described in the second aspect; the control plane device for determining the number of global replica groups activated in each training step of the target training; the training devices for obtaining the number of replica groups from the control plane device and determining the total data parallelism scale through the means for controlling the global batch size; and the means for controlling the global batch size for dynamically adjusting training hyperparameters according to the expected GBS and the total data parallelism scale, so that the GBS value of each training step is consistent with the expected GBS value.
[0029] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method of controlling the global batch size of the first aspect.
[0030] Fifthly, a computer device including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the computer-readable instructions, when executed by the processor, implement the method of controlling global batch size as described in the first aspect.
[0031] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: In distributed deep learning training, if elastic training or fault-tolerant training is performed, the expected GBS during the training process can be obtained. Before each training step begins, the total parallel scale of the current training data is dynamically calculated. Based on the expected GBS and the total parallel scale, the training hyperparameters are dynamically adjusted so that the GBS value of each training step is consistent with the expected GBS value. During the training process, there are situations where devices dynamically join or leave. When the total parallel scale of the current training data changes due to changes in the number of training nodes caused by elastic training or fault-tolerant training, the training hyperparameters can be adjusted based on the current total parallel scale of the current training data and the expected GBS value of the training step. This allows the adjusted training hyperparameters to maintain the expected GBS, reducing or even suppressing GBS fluctuations when devices dynamically join or leave. This reduces problems such as gradient noise, gradient failure, and learning rate failure caused by excessive GBS fluctuations, ultimately reducing the phenomenon of decreased model training stability and convergence, thereby reducing the probability of training failure and improving the accuracy and stability of model training.
[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0033] The accompanying drawings, which are incorporated in and form part of this disclosure, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0034] Figure 1 This is a schematic diagram of a system for controlling the global batch size, provided as an embodiment of the present disclosure.
[0035] Figure 2 This is a flowchart illustrating a method for controlling the global batch size provided in an embodiment of the present disclosure.
[0036] Figure 3 This is a schematic diagram of a multi-card pipeline training process provided in an embodiment of the present disclosure.
[0037] Figure 4 This is a schematic diagram of a data flow for controlling the global batch size, provided as an embodiment of the present disclosure.
[0038] Figure 5 This is a hardware structure diagram of a computer device containing a means for controlling the global batch size, as provided in an embodiment of this disclosure.
[0039] Figure 6 This is a schematic diagram of a device structure for controlling the global batch size, provided as an embodiment of the present disclosure. Detailed Implementation
[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0041] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.
[0042] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0043] The embodiments of this disclosure will now be described in detail.
[0044] like Figure 1 As shown, Figure 1 This is a schematic diagram of a system for controlling the global batch size provided in an embodiment of the present disclosure, such as... Figure 1 The diagram shows a control plane device 101 and one or more training devices 102 participating in the target training. Each training device 102 includes a device for controlling the Global Replica Groups (GBS). The target training is either elastic or fault-tolerant training within distributed training, and devices may dynamically join or leave during the target training process. The control plane device 101 determines the number of global replica groups activated in each training step. Before each training step begins, the training device 102 dynamically calculates the total parallel scale of the currently participating data based on the number of replica groups obtained from the control plane device, using the GBS control device. The GBS control device in each training device 102 dynamically adjusts the training hyperparameters based on the expected GBS and the total parallel scale, ensuring that the GBS value in each training step matches the expected GBS value.
[0045] It is understandable that each training device adjusts its local training hyperparameters based on the global total data parallelism and GBS.
[0046] Figure 2 This is a flowchart illustrating a method for controlling the global batch size according to an embodiment of the present disclosure, as shown below. Figure 2 As shown, steps S201 to S203 are included.
[0047] S201. Obtain the expected GBS during the target training process.
[0048] Among them, target training is elastic training or fault-tolerant training in distributed deep learning training, and there are dynamic additions or exits of devices during the target training process.
[0049] Specifically, resilient training refers to the ability to dynamically adjust the number of workers (devices or workloads (GPUs)) participating in training during distributed training to adapt to resource changes or fault recovery. Fault-tolerant training is a machine learning method that actively introduces or simulates faults in internal components (such as neurons and connection weights) during model training, enabling the model to maintain stable performance and accurate output even when its structure is partially damaged or computational errors occur.
[0050] Alternatively, GBS can be user-specified or set as the default during training.
[0051] Optionally, if the expected GBS specified by the user can be directly obtained in a training step, the directly obtained expected GBS is used. If the expected GBS specified by the user cannot be directly obtained in a training step, and the user has defined GBS change rules, the expected GBS in that training step is calculated in real time based on the defined GBS change rules.
[0052] S202. Before each training step begins, dynamically calculate the total parallel scale of the data currently participating in the training.
[0053] The total data parallel scale indicates the total number of devices participating in the parallel training of distributed deep learning, with each device corresponding to one process.
[0054] For example, the devices involved in parallel training can be GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), or CPUs (Central Processing Units).
[0055] During elastic or fault-tolerant training, the total data parallelism scale often changes. The total data parallelism scale participating in training can be statistically analyzed through the control plane of large-scale distributed elastic or fault-tolerant training, summarizing the total data parallelism scale participating in each training step at the beginning of each training step.
[0056] S203. Based on the expected GBS and the total parallel scale of the data, dynamically adjust the training hyperparameters so that the GBS value of each training step is consistent with the expected GBS value.
[0057] Training hyperparameters refer to the configuration parameters set before the model begins training. These training hyperparameters are not learned from the data, but rather used to control the behavior and performance of the training process itself.
[0058] For example, training hyperparameters can be the number of micro-batch sizes and the number of gradient accumulations.
[0059] This disclosure provides a method for controlling the global batch size. In distributed deep learning training, if elastic training or fault-tolerant training is performed, the expected GBS (Global Batch Size) during training can be obtained. Before each training step, the total data parallelism scale currently participating in training is dynamically calculated. Based on the expected GBS and the total data parallelism scale, the training hyperparameters are dynamically adjusted so that the actual GBS value of each training step is consistent with the expected GBS value. During training, devices may dynamically join or leave. If the total data parallelism scale currently participating in training changes due to changes in the number of training nodes caused by elastic or fault-tolerant training, the training hyperparameters can be adjusted based on the current total data parallelism scale and the expected GBS value of the training step. This allows the adjusted training hyperparameters to maintain the expected GBS, reducing or even suppressing GBS fluctuations when devices dynamically join or leave. This reduces problems such as gradient noise, gradient failure, and learning rate failure caused by excessive GBS fluctuations, ultimately reducing the decrease in model training stability and convergence, thereby reducing the probability of training failure and improving the accuracy and stability of model training.
[0060] Optionally, in the method for controlling the global batch size provided in this embodiment of the disclosure, the above-mentioned S201 can be specifically executed through the following S201a: S201a. Before each training step, determine the expected GBS for the current training step based on the increase method specified by the target strategy used during training, the total target GBS, and the expected GBS of the previous training step.
[0061] For example, the target policy can be a progressively increasing ramp-up policy or a user-defined GBS increment policy. The target policy can indicate the increment of GBS for each training step compared to the previous training step. This can be indicated based on a custom function.
[0062] Based on this scheme, the training system can accurately determine the expected GBS for each training step according to the parameters specified in the GBS variation strategy used during training. This allows the training hyperparameters to be adjusted based on the expected GBS value to adapt to the expected GBS, thereby reducing GBS fluctuations during training caused by changes in the number of nodes and the failure of the training hyperparameters to adapt.
[0063] Optionally, in the method for controlling the global batch size provided in the embodiments of this disclosure, if the target policy is a ramp-up policy, indicating that the batch size is gradually increased from the first GBS to the second GBS after N training steps, it can be executed through the following S201a1.
[0064] S201a1. Determine the GBS increment for each training step based on the first GBS, the second GBS, and the total number of training steps. Determine the expected GBS for the current training step based on the GBS increment and the number of steps in the current training step.
[0065] In practical applications, the GBS increment between any two adjacent training steps in a target policy can be the same or different. It's understandable that the GBS increment between any two adjacent training steps is the same in a ramp-up policy.
[0066] For example, when using the Ramp-up strategy, the expected GBS for the N1 training step can be determined based on formula (1).
[0067] GBS N1 =GBS1+(GBS2-GBS1)×(N1-1) / N Formula (1) Among them, GBS N1 Let GBS1 represent the expected GBS for the N1 training step, GBS2 represent the first GBS, and GBS3 represent the second GBS.
[0068] Based on this scheme, if the Ramp-up strategy is adopted, each training step has a desired GBS value, and the expected GBS within the current training step cycle can be determined according to the indication information in the Ramp-up strategy and the current number of training steps.
[0069] Optionally, in the method for controlling the global batch size provided in the embodiments of this disclosure, the above-mentioned S202 includes S202a and S202b.
[0070] S202a. Before each training step begins, obtain the number of currently active copy groups.
[0071] Typically, in engineering, fault-tolerant and resilient training schemes involve splitting the system into multiple replica groups, with a control plane used to statistically coordinate these groups.
[0072] In this context, a replica is a complete copy of the model. In single-data parallelism, there is one complete model on one GPU, and the model on each GPU is a replica. In hybrid parallelism, multiple GPUs share a complete copy, and the model formed by these multiple GPUs constitutes a replica.
[0073] A replica group refers to a collection of all devices (or processes) that jointly hold and collaboratively maintain a complete replica of a model, typically with each process controlling one GPU.
[0074] A single training task is divided into multiple training sub-tasks. Each replica group corresponds to one of these sub-tasks. While the data types within each replica group are the same, the data they process differs. All replica groups collectively represent the same project. Each replica group processes separate data, and the different replica groups operate independently of each other, synchronizing only gradient values.
[0075] S202b. Determine the total data parallel scale currently participating in training based on the number of replica groups and the data parallel scale of a single replica group.
[0076] The data parallelism scale of a single replica group indicates the number of processes or GPUs performing data parallelism within a replica group.
[0077] Specifically, at the start of each training step, the number of currently active replica groups (ReplicaNums) is obtained from the control plane.
[0078] If the data parallel size of each replica group is constant, then the total data parallel size = the number of replica groups × the data parallel size of a single replica group.
[0079] If the data parallelism scale is different in each replica group, then the total data parallelism scale is equal to the sum of the products of the data parallelism scale of each replica group and the corresponding individual replica group.
[0080] Based on this scheme, the total data parallelism involved in training during that training step can be accurately obtained before the start of each training step, based on the number of replica groups and the data parallelism scale of a single replica group.
[0081] Optionally, in the method for controlling the global batch size provided in the embodiments of this disclosure, the above-mentioned S203 can be specifically executed through S203a.
[0082] S203a: Based on the expected GBS, total data parallel scale, and model segmentation for target training, dynamically adjust the training hyperparameters.
[0083] The model segmentation status indicates whether the model has been segmented, and the specific training hyperparameters that need to be adjusted can be determined based on the model segmentation status.
[0084] Optionally, in the method for controlling the global batch size provided in the embodiments of this disclosure, the above-mentioned S203 further includes the following S203b and S203c.
[0085] S203b, Determine the model segmentation index for target training.
[0086] The model splitting index is used to indicate whether pipelined parallelism exists during the training process.
[0087] Specifically, the model splitting index can be configured in the distributed strategy configuration.
[0088] For example, the Degree of pipeline model parallelism in the program corresponds to the parameter name: pipeline-model-parallel-size.
[0089] The model split index equal to 1 indicates that pipelined parallelism is not used in the training process, while the model split index greater than 1 indicates that pipelined parallelism is used in the model training process.
[0090] S203c. Based on the model segmentation index, determine the training hyperparameters to be adjusted for the target training.
[0091] The different model segmentation indices result in different training hyperparameters.
[0092] It should be noted that since different distributed strategies affect the training hyperparameters of GBS differently, the training hyperparameters to be adjusted during training can be determined based on the model splitting index before adjustment.
[0093] Based on this scheme, the type of training hyperparameters to be adjusted can be determined according to the pipeline parallel strategy adopted by elastic training or fault-tolerant training, so that training hyperparameters that match the actual training can be accurately selected for adjustment.
[0094] Optionally, in the method for controlling the global batch size provided in the embodiments of this disclosure, S203 may specifically include the following S203d.
[0095] S203d. When the model splitting index is greater than 1, the training hyperparameter to be adjusted for the target training is determined to be the number of micro-batches. Based on the expected GBS, micro-batch size and total data parallel scale, the adjusted number of micro-batches is determined.
[0096] Specifically, the number of micro-batches can be determined based on formula (2).
[0097] Micro-batch size = Expected GBS / / (micro-batch size × total data parallel scale) Formula (2) It should be noted that if pipelined parallelism is used, then in target training, the number of micro-batches (Num MicroBatches) can represent the number of forward and backward passes processed by each card in one cycle.
[0098] Based on this scheme, when using pipeline parallelism, the number of micro-batches, representing the number of forward and backward passes processed by each card in one cycle, can be determined based on the expected GBS, micro-batch size, and total data parallelism scale during elastic training or fault-tolerant training. Thus, the pipeline length of the current training step can be adjusted according to the number of micro-batches to adjust the length of the pipeline data consumed, thereby maintaining the stability of GBS.
[0099] Optionally, in the method for controlling the global batch size provided in the embodiments of this disclosure, S203 may specifically include the following S203e.
[0100] S203e. When the model splitting exponent is equal to 1, the training hyperparameter to be adjusted for the target training is the gradient accumulation number. Based on the expected GBS, micro-batch size and total data parallel scale, the adjusted micro-batch number is determined.
[0101] Specifically, the number of gradient accumulations can be determined based on formula (3).
[0102] Gradient accumulation count = expected GBS / / (micro-batch size × total data parallel scale) Formula (3) If pipelined parallelism is not used, the number of gradient accumulations (Grad Accumulate Nums) during target training can represent the number of forward and backward passes processed by each card in one cycle.
[0103] For example, each training step may include: forward, backward, and update. In a pipelined scenario, each training step includes (forward, backward), ..., (forward, backward), and update. The number of micro-batches can represent the number of (forward, backward) operations in each training step in a pipelined parallel scenario. The number of gradient accumulations can represent the number of (forward, backward) operations in each training step in a non-pipeline parallel scenario.
[0104] Based on this scheme, without using pipelined parallelism, the gradient accumulation count, which represents the number of forward and backward passes processed by each card in one cycle during elastic training or fault-tolerant training, can be determined based on the expected GBS, micro-batch size, and total data parallelism scale. Thus, the number of forward and backward passes in the current training step can be adjusted according to the gradient accumulation count to adjust the length of the training data consumed, thereby maintaining the stability of GBS.
[0105] Optionally, in the method for controlling the global batch size provided in the embodiments of this disclosure, the above-mentioned S203d may specifically include Sd1 or Sd2 as described below.
[0106] Sd1. If at the start of training, the number of replica groups participating in training is the number of the first replica groups, and the data parallel scale of each replica group is the first data parallel scale, then based on the number of the first replica groups and the first data parallel scale, the dynamic total data parallel scale of the first training step is determined, and based on the first GBS micro-batch size and the dynamic total data parallel scale of the first training step, the number of micro-batches within the first training step cycle is determined.
[0107] For example, based on formula (4), the dynamic total data parallel scale of the first training step is determined, and based on formula (5), the number of micro-batches in the first training step cycle is determined.
[0108] TPS1=RN1×DPS1 formula (4) Where TPS1 represents the dynamic total data parallel scale of the first training step, RN1 represents the number of the first replica groups, and DPS1 represents the first data parallel scale.
[0109] NMB 1= GBS1 / / (MB*TPS1) formula (5) Where NMB1 represents the number of micro-batches in the first training step cycle, and MB represents the micro-batch size, which usually remains unchanged during training.
[0110] It should be noted that the number of gradient accumulations can also be determined by referring to the above formula (4).
[0111] Sd2. If, before training begins in the N1st training step, the number of activated replica groups becomes the number of second replica groups, then based on the number of second replica groups and the first data parallel scale, determine the total data parallel scale of the N1st training step, and based on the expected GBS, micro-batch size, and total data parallel scale of the N1st training step, determine the number of micro-batches within the N1st training step period.
[0112] For example, based on formula (6), the total data parallel scale of the N1 training step is determined, and based on formula (7), the number of micro-batches within the N1 training step cycle is determined.
[0113] TPS N1 =RN2×DPS1 formula (6) Among them, TPS N1 RN1 represents the total parallel scale of the data in the N1th training step, and RN2 represents the number of the second copy groups.
[0114] NMB N1 =GBS N1 / / (MB*TPS) N1 ) Formula (7) Among them, NMB N1 This represents the number of micro-batches in the N1th training step.
[0115] Based on this scheme, when using a ramp-up strategy and pipeline parallelism during elastic or fault-tolerant training, the dynamic total data parallelism scale of each training step can be determined based on the number of replica groups that can be activated in each training step and the data parallelism scale of each replica group. Then, based on the dynamic total data parallelism scale of each training step and the dynamic GBS of each training step, the number of micro-batches in each training step cycle can be determined, and the number of micro-batches in model training can be adjusted based on the determined number of micro-batches in each training step cycle.
[0116] It should be noted that the number of gradient accumulations can also be determined by referring to the above formulas (4) and (6).
[0117] Optionally, in the method for controlling the global batch size provided in this embodiment of the disclosure, in S203b above, the number of micro-batches in the current training step is smaller than that in the previous training step, that is, the pipeline length of the current training step is shortened. Alternatively, the number of micro-batches in the current training step is larger than that in the previous training step, that is, the pipeline length of the current training step is increased.
[0118] For example, if the control plane detects a decrease in the number of replica groups, determines that the total data parallelism scale has decreased, and further determines that the number of micro-batches has decreased, that is, the number of micro-batches in the current training step is smaller than the number of micro-batches in the previous training step, then the number of forward and backward passes is adjusted according to the determined number of micro-batches in the current training step. For example, if the previous training step included K1 (forward and backward passes), the current training step reduces it to K2 (forward and backward passes). The pipeline length becomes shorter, so less data is consumed during training, in order to keep the actual GBS value close to the expected GBS value.
[0119] For example, if the control plane detects an increase in the number of replica groups, determines that the total data parallelism scale has increased, and further determines that the number of micro-batches has increased, that is, the number of micro-batches in the current training step is larger than the number of micro-batches in the previous training step, then the number of forward and backward passes is adjusted according to the determined number of micro-batches in the current training step. For example, if the previous training step included K1 (forward and backward passes), and the current training step increases to K3 (forward and backward passes), the pipeline length becomes longer, and more data is consumed during training in order to keep the actual GBS value close to the expected GBS value.
[0120] For example, Figure 3 This is a schematic diagram of a multi-card pipeline training process provided in an embodiment of the present disclosure, such as... Figure 3As shown, in the multi-card pipeline training process, the training model is divided into multiple layers along its length. For example, the first card is used to train the first and second layers of the model, the second card is used to train the third and fourth layers, and the third card is used to train the fifth and sixth layers. The first card calculates output 1 based on microbatch 1 and inputs output 1 into the second card, while continuing to calculate based on microbatch 2. The second card calculates output 2 based on output 1 and inputs it into the third card. The third card calculates output 3 and loss 1 based on output 2 and inputs loss 1 into the second card. The second card inputs loss 1 into the first card. The idle time length of each card is fixed. If the pipeline is longer, the idle time length percentage becomes smaller, for example, from 10% to 5%, resulting in higher card resource utilization.
[0121] Based on this scheme, the length of the training pipeline can be adjusted according to the calculated number of micro-batches, so that the actual GBS value is close to the expected GBS value when training with a new number of replicas. Furthermore, since large model training generally uses pipeline parallelism, the number of available training nodes typically decreases gradually during fault-tolerant training. This increases the number of micro-batches, which helps reduce waiting time during pipeline parallelism and improves the utilization of GPU resources.
[0122] Optionally, in S203c of the method for controlling the global batch size provided in this embodiment of the disclosure, the number of gradient accumulations in the current training step is smaller than that in the previous training step, that is, the number of forward-backward loops is reduced. Conversely, the number of gradient accumulations in the current training step is larger than that in the previous training step, that is, the number of forward-backward loops is increased.
[0123] Similarly, if the control plane detects a decrease in the number of replica groups, determines that the total data parallelism scale and the number of gradient accumulations have decreased, it can be determined that the number of gradient accumulations in the current training step should be smaller than the number of gradient accumulations in the previous training step. Then, the number of forward and backward iterations is adjusted according to the determined number of gradient accumulations in the current training step, so that less data is consumed during training, in order to keep the actual GBS value close to the expected GBS value.
[0124] Similarly, if the control plane detects an increase in the number of replica groups, and determines that the total data parallelism and gradient accumulation count have increased, it can be determined that the gradient accumulation count in the current training step should be greater than that in the previous training step. Then, the forward and backward iteration counts are adjusted according to the determined gradient accumulation count in the current training step, resulting in more data being consumed during training to keep the actual GBS value close to the expected GBS value.
[0125] Based on this scheme, without using pipelined parallelism, the number of forward-backward loops during training can be adjusted according to the calculated number of gradient accumulations, so that the actual GBS value is close to the expected GBS value when training with a new number of replicas.
[0126] Figure 4 This is a schematic diagram illustrating the data flow for maintaining expected GBS stability during distributed training, as provided in an embodiment of this disclosure. Figure 4 As shown, the expected GBS for the training step is determined based on the configuration of the final GBS and the ramp-up strategy; the control plane determines the number of replica groups and the total data parallelism of the training step is determined based on the local data parallelism of the replica groups (i.e., the data parallelism of a single replica group); the micro-batch size or the number of gradient accumulations is determined based on the expected GBS for the training step and the total data parallelism, thereby adjusting the training hyperparameters during training.
[0127] Example Step 1: A multi-GPU pipelined parallel distributed training task is started normally, with micro-batch size set to 1 and GBS set to 1024. A ramp-up strategy is used to dynamically increase GBS from 128 to 1024 over 8 training steps.
[0128] Step 2: At the start of training, machine resources may be added gradually. Let's assume the number of training replica groups is 4. The local data parallelism scale for each replica is assumed to be 8, a value set by the user at startup. Therefore, the dynamic total data parallelism scale at this point = 4 * 8 = 32.
[0129] Step 3: Based on the dynamic GBS=128, the number of micro-batches in this training step cycle is calculated as 128 / 32=4.
[0130] Step 4: When the training reaches the 5th training cycle, the GBS becomes (1024-128)*4 / 8+128=576.
[0131] Step 5: Assume the number of replica groups participating in training now becomes 8, meaning more computing resources are allocated to training. The total dynamic data parallelism scale at this point is 8 * 8 = 64.
[0132] Step 6: The final number of micro-batches for this 5th training cycle is 576 / / 64=9.
[0133] Corresponding to the embodiments of the foregoing methods, this disclosure also provides embodiments of the apparatus and the terminal to which it is applied.
[0134] Embodiments of the method for controlling the global batch size disclosed herein can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by a processor controlling the global batch size loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5The diagram shown is a hardware structure diagram of a computer device containing a device for controlling the global batch size, as provided in an embodiment of this disclosure. Except for... Figure 5 In addition to the processor 510, memory 530, network interface 520, and non-volatile memory 540 shown, the server or electronic device where the system 531 controlling the global batch size in the embodiment is located may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.
[0135] Figure 6 This is a schematic diagram of a device structure for controlling the global batch size according to an embodiment of the present disclosure. The device 600 for controlling the global batch size includes: a GBS acquisition unit 601, a data scale acquisition unit 602, and a hyperparameter adjustment unit 603. The GBS acquisition unit 601 is used to acquire the expected GBS during the target training process. The target training is elastic training or fault-tolerant training in distributed training, and the target training involves dynamic addition or removal of devices. The data scale acquisition unit 602 is used to dynamically calculate the total parallel scale of the data currently participating in the training before the start of each training step. The hyperparameter adjustment unit 603 is used to dynamically adjust the training hyperparameters based on the expected GBS and the total parallel scale of the data, so that the actual GBS value of each training step is consistent with the expected GBS value.
[0136] Optionally, the data scale acquisition unit is specifically used to: before the start of each training step, acquire the number of currently active replica groups, where a replica group indicates a set of devices that jointly maintain a replica of a model; and determine the total data parallelism scale currently participating in training based on the number of replica groups and the data parallelism scale of a single replica group.
[0137] Optionally, a hyperparameter tuning unit is used to dynamically adjust training hyperparameters based on the expected GBS, total data parallelism, and model splitting for target training.
[0138] Optionally, the device for controlling the global batch size further includes: a model split index determination unit; the model split index determination unit is used to determine the model split index for the target training, the model split index being used to indicate whether pipeline parallelism exists during the training process; and a hyperparameter adjustment unit is also used to determine the training hyperparameters to be adjusted for the target training based on the model split index; wherein, different model split indices result in different training hyperparameters.
[0139] Optionally, the hyperparameter tuning unit is specifically used to: determine the target training hyperparameter to be tuned as the number of micro-batches when the model splitting exponent is greater than 1, and determine the adjusted number of micro-batches based on the expected GBS, the micro-batch size and the total data parallel scale.
[0140] Optionally, the hyperparameter tuning unit is specifically used to: determine the number of gradient accumulations as the training hyperparameter to be tuned for the target training when the model splitting exponent is equal to 1, and determine the adjusted number of gradient accumulations for the target training based on the expected GBS, micro-batch size and total data parallel scale.
[0141] Optionally, the number of micro-batches in the current training step is smaller than that in the previous training step, i.e., the pipeline length of the current training step is shortened; or the number of micro-batches in the current training step is larger than that in the previous training step, i.e., the pipeline length of the current training step is increased.
[0142] Optionally, the number of gradient accumulations in the current training step is smaller than that in the previous training step, that is, the number of forward-backward loops in the current training step is reduced; or the number of gradient accumulations in the current training step is larger than that in the previous training step, that is, the number of forward-backward loops in the current training step is increased.
[0143] Optionally, the GBS acquisition unit is specifically used to: before each training step, determine the expected GBS for the current training step based on the GBS increment method specified by the target strategy adopted for training, the total target GBS, and the expected GBS of the previous training step.
[0144] Optionally, the target strategy is a ramp-up strategy that indicates a gradual increase from the first GBS to the second GBS over N training steps; the GBS acquisition unit is specifically used to: determine the GBS increment for each training step based on the first GBS, the second GBS, and the total number of training steps, and determine the expected GBS for the current training step based on the GBS increment and the number of steps in the current training step.
[0145] Optionally, the hyperparameter tuning unit is specifically used to: if at the start of training, the number of replica groups participating in training is the number of the first replica groups, and the data parallelism scale of each replica group is the first data parallelism scale, then based on the number of the first replica groups and the first data parallelism scale, determine the dynamic total data parallelism scale of the first training step, and based on the first GBS, the micro-batch size, and the dynamic total data parallelism scale of the first training step, determine the number of micro-batches in the first training step cycle; if before the start of training in the N1th training step, the number of activated replica groups becomes the number of the second replica groups, then based on the number of the second replica groups and the first data parallelism scale, determine the total data parallelism scale of the N1th training step, and based on the expected GBS, the micro-batch size, and the total data parallelism scale of the N1th training step, determine the number of micro-batches in the N1st training step cycle.
[0146] This disclosure provides a device for controlling the global batch size. In distributed deep learning training, if elastic training or fault-tolerant training is performed, the expected GBS during the training process can be obtained. Before each training step begins, the total data parallelism scale currently participating in the training is dynamically calculated. Then, based on the expected GBS and the total data parallelism scale, the training hyperparameters are dynamically adjusted so that the actual GBS value of each training step is consistent with the expected GBS value. During training, devices may dynamically join or leave. If the total data parallelism scale currently participating in the training changes due to changes in the number of training nodes caused by elastic or fault-tolerant training, the training hyperparameters can be adjusted based on the current total data parallelism scale and the expected GBS value of the training step. This allows the adjusted training hyperparameters to maintain the expected GBS, reducing or even suppressing GBS fluctuations when devices dynamically join or leave. This reduces problems such as gradient noise, gradient failure, and learning rate failure caused by excessive GBS fluctuations, ultimately reducing the decrease in model training stability and convergence, thereby reducing the probability of training failure and improving the accuracy and stability of model training.
[0147] This disclosure provides a large-scale distributed deep learning system, comprising: a control plane device and one or more training devices participating in target training, and the training devices including, for example, Figure 6 The device for controlling the global batch size; a control plane device for determining the number of global replica groups activated in each training step of the target training; a training device for obtaining the number of replica groups from the control plane device and determining the total data parallel scale through the device for controlling the global batch size; and a device for controlling the global batch size for dynamically adjusting the training hyperparameters according to the expected GBS and the total data parallel scale so that the GBS value of each training step is consistent with the expected GBS value.
[0148] This disclosure also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for controlling the global batch size as described in the above method embodiments.
[0149] This disclosure also provides a computer device, the computer device including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the computer-readable instructions, when executed by the processor, implement the method for controlling the global batch size as described in the above method embodiments.
[0150] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0151] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0152] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0153] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention applied herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0154] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
[0155] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A method for controlling the global batch size, characterized in that, The method includes: Obtain the expected global batch size GBS during the target training process, wherein the target training is elastic training or fault-tolerant training in distributed deep learning training, and the target training process involves dynamic addition or removal of devices. Before each training step begins, dynamically calculate the total parallel scale of the current data participating in training. Based on the expected GBS and the total parallel data scale, the training hyperparameters are dynamically adjusted so that the GBS value of each training step is consistent with the expected GBS value.
2. The method according to claim 1, characterized in that, Before each training step begins, the total parallel scale of the currently participating data is dynamically calculated, including: Before each training step begins, obtain the number of currently active replica groups, where a replica group indicates the set of devices that jointly maintain a replica of a model; The total data parallelism scale currently participating in training is determined based on the number of replica groups and the data parallelism scale of a single replica group.
3. The method according to claim 1, characterized in that, The dynamic adjustment of training hyperparameters based on the expected GBS and the total parallel data scale includes: The training hyperparameters are dynamically adjusted based on the expected GBS, the total parallel data scale, and the model segmentation for the target training.
4. The method according to claim 3, characterized in that, The dynamic adjustment of training hyperparameters based on the expected GBS and the total parallel data scale includes: Determine the model split index for the target training, which is used to indicate whether pipeline parallelism exists during the training process; The training hyperparameters for the target training are determined based on the model segmentation index, wherein different model segmentation indices result in different training hyperparameters.
5. The method according to claim 4, characterized in that, The dynamic adjustment of training hyperparameters based on the expected GBS and the total parallel data scale includes: If the model segmentation index is greater than 1, the training hyperparameter to be adjusted for the target training is determined to be the number of micro-batches. The adjusted number of micro-batches is determined based on the expected GBS, the micro-batch size, and the total parallel scale of the data.
6. The method according to claim 4, characterized in that, The dynamic adjustment of training hyperparameters based on the expected GBS and the total parallel data scale includes: When the model splitting exponent is equal to 1, the training hyperparameter to be adjusted for the target training is determined to be the gradient accumulation count. Based on the expected GBS, micro-batch size and total data parallel scale, the adjusted gradient accumulation count for the target training is determined.
7. The method according to claim 1, characterized in that, The acquisition of the expected GBS during the target training process includes: Before each training step, the expected GBS for the current training step is determined based on the GBS increment method specified by the target strategy adopted for training, the total target GBS, and the expected GBS of the previous training step.
8. The method according to claim 7, characterized in that, The target strategy is the Ramp-up strategy, which indicates that the GBS is gradually increased from the first GBS to the second GBS through N training steps. Determining the expected GBS for the current training step includes: The GBS increment for each training step is determined based on the first GBS, the second GBS, and the total number of training steps. The expected GBS for the current training step is determined based on the GBS increment and the number of steps in the current training step.
9. The method according to claim 5, characterized in that, When the model's segmentation index is greater than 1, the number of micro-batches is determined based on the dynamic global batch size, micro-batch size, and total data parallelism scale, including: If at the start of training, the number of replica groups participating in training is the number of the first replica groups, and the data parallel scale of each replica group is the first data parallel scale, then based on the number of the first replica groups and the first data parallel scale, the dynamic total data parallel scale of the first training step is determined, and based on the first GBS, the micro-batch size, and the dynamic total data parallel scale of the first training step, the number of micro-batches within the first training step cycle is determined. If, before training begins in the N1st training step, the number of activated replica groups becomes the number of second replica groups, then based on the number of second replica groups and the first data parallel scale, the total data parallel scale of the N1st training step is determined, and based on the expected GBS of the N1st training step, the micro-batch size, and the total data parallel scale of the N1st training step, the number of micro-batches within the N1st training step period is determined.
10. A device for controlling the global batch size, characterized in that, The device includes: a global batch size (GBS) acquisition unit, a data scale acquisition unit, and a hyperparameter adjustment unit; The GBS acquisition unit is used to acquire the expected GBS during the target training process. The target training is elastic training or fault-tolerant training in distributed training. During the target training process, there are situations where devices dynamically join or leave. The data scale acquisition unit is used to dynamically calculate the total parallel scale of the data currently participating in training before each training step begins. The hyperparameter adjustment unit is used to dynamically adjust the training hyperparameters based on the expected GBS and the total parallel data scale, so that the GBS value of each training step is consistent with the expected GBS value.
11. A large-scale distributed deep learning system, characterized in that, The system includes: a control plane device and one or more training devices participating in target training, and the training devices include the means for controlling the global batch size as described in claim 10; The control plane device is used to determine the number of global replica groups activated in each training step for the target training. The training device is used to obtain the number of replica groups from the control plane device and to determine the total data parallelism scale by means of a device that controls the global batch size; A device for controlling the global batch size, used to dynamically adjust training hyperparameters based on the expected GBS and the total parallel data scale, so that the GBS value of each training step is consistent with the expected GBS value.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for controlling the global batch size as described in any one of claims 1-9.
13. A computer device, characterized in that, The computer device includes a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the computer-readable instructions, when executed by the processor, implement the method for controlling the global batch size as described in any one of claims 1-9.