Deep learning training method based on communication masking rate and related device

Abstract

The application is suitable for the technical field of distributed deep learning training, and provides a deep learning training method based on communication cover rate, which comprises the following steps: after a training task is started, entering a configuration stage in the first K training steps, obtaining multiple groups of candidate training configurations and obtaining the communication cover rate corresponding to each group of candidate training configurations, the candidate training configurations being used for configuring multiple training parameters required by the training task; based on the communication cover rate, a training step time consumption model containing multiple training parameters is constructed; a search space is determined according to the multiple groups of candidate training configurations, and multiple training parameters are jointly optimized through structured grid search under a preset constraint to find a configuration combination maximizing the effective throughput, so that the optimal training configuration of the training task is obtained; the optimal training configuration is used to configure the training parameters for the training task, the configuration stage is completed, the training stage is entered, and the training parameters are adjusted in real time by tracking the change of the real-time communication cover rate.

Description

Technical Field

[0001] This application belongs to the field of distributed deep learning training technology, and in particular relates to a deep learning training method and related equipment based on communication masking rate. Background Technology

[0002] Data parallelism is the most widely adopted parallel strategy for large-scale deep learning training. In data parallelism, each acceleration core holds a complete copy of the model, processes different batches of data, and synchronizes gradients through ensemble communication operations such as AllReduce after each training step. The end-to-end time of a training step determines the system's training throughput.

[0003] Ideally, gradient communication and backpropagation computation can completely overlap—communication begins immediately after the gradient computation of the later layer is completed, executing in parallel with the gradient computation of the previous layer. If communication can be completely "masked" by computation, then communication does not add any additional latency, and the training step time equals the pure computation time. However, in real-world systems, the degree of overlap between communication and computation is constrained by various factors and is not simply a matter of "complete overlap" or "complete non-overlap."

[0004] Currently, existing distributed training frameworks have significant shortcomings in performance modeling and configuration optimization. Performance models either assume complete overlap between communication and computation (as in the design goal of PyTorch DDP) or complete non-overlap (as in the simple time-consuming analysis formula T = T_comp + T_comm). Both assumptions are rough approximations of the real situation, resulting in a lack of accurate performance models for configuration optimization. Existing automatic parameter tuning methods (such as automatic learning rate search and batch size adjustment) usually only adjust a single parameter, while the local batch size, gradient accumulation steps, and communication bucket size together determine the time structure of computation and communication, requiring joint optimization. The communication masking rate is affected by factors such as runtime bus load, resource contention of other processes, and hardware thermal state, and is a dynamically changing value. Static one-time parameter tuning cannot adapt to this change. Therefore, there is an urgent need for a deep learning training method based on communication masking rate, which uses communication masking rate as the core parameter and adaptively optimizes local batch size, gradient accumulation steps, and communication bucket size. This would address the problems of existing methods, where the performance model either assumes complete overlap or complete non-overlap, both of which deviate significantly from reality. Parameter tuning is usually limited to a single dimension, and the methods do not jointly optimize local batch size, gradient accumulation steps, and communication bucket size, thus lacking the ability to adapt to changes in the runtime environment. Summary of the Invention

[0005] This application provides a deep learning training method based on communication masking rate, which addresses the problems of existing methods where performance models either assume complete overlap or complete non-overlap, both of which deviate significantly from reality. Parameter tuning is typically limited to a single dimension, failing to jointly optimize local batch size, gradient accumulation steps, and communication bucket size, and lacking adaptability to changes in the runtime environment. This method constructs a training step time model containing multiple training parameters using the communication masking rate corresponding to each set of candidate training configurations. A search space is determined based on multiple candidate training configurations, and under preset constraints, multiple training parameters are jointly optimized using structured grid search to find the configuration combination that maximizes effective throughput, obtaining the optimal training configuration for the training task. The optimal training configuration is then used to configure the training parameters for the training task, completing the configuration phase and entering the training phase. By tracking real-time changes in communication masking rate, the training parameters are hot-tuned. This solves the problems of existing methods where performance models either assume complete overlap or complete non-overlap, both of which deviate significantly from reality. Parameter tuning is typically limited to a single dimension, failing to jointly optimize local batch size, gradient accumulation steps, and communication bucket size, and lacking adaptability to changes in the runtime environment.

[0006] In a first aspect, embodiments of this application provide a deep learning training method based on communication masking rate, the method comprising the following steps:

[0007] After the training task starts, the configuration phase is entered in the first K training steps. Multiple sets of candidate training configurations are obtained, and for each set of candidate training configurations, the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side are collected. Based on the computation time, communication time, and combined training time, the communication masking rate corresponding to each set of candidate training configurations is calculated. The candidate training configuration is used to configure multiple training parameters required by the training task. The communication masking rate is used to quantify the degree of overlap between computation and communication in the training task. The computation time is collected by disabling AllReduce callbacks on the pure computation side. The communication time is collected by skipping computation on the pure communication side and only executing AllReduce callbacks on tensors of the same size as the gradient. The combined training time is collected by normal training on the computation and communication side.

[0008] Based on the communication masking rate, a training step time model containing multiple training parameters is constructed;

[0009] The search space is determined based on multiple sets of candidate training configurations, and under preset constraints, multiple training parameters are jointly optimized through structured grid search to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0010] The optimal training configuration is used to configure the training parameters for the training task, completing the configuration phase and entering the training phase. The training parameters are then hot-tuned by tracking changes in the real-time communication masking rate.

[0011] Optionally, the step of calculating the communication masking rate corresponding to each group of candidate training configurations based on the computation time, communication time, and overall training time includes:

[0012] Based on the computation time and the comprehensive training time, the additional latency actually exposed by the communication is calculated.

[0013] Based on the additional latency and the communication time, the communication masking rate corresponding to each group of candidate training configurations is calculated.

[0014] Optionally, constructing a training step time model containing multiple training parameters based on the communication masking rate includes:

[0015] Based on the communication masking rate, calculate the total computation time for step gradient accumulation, the residual delay of unmasked communication, and the fixed overhead of parameter updates;

[0016] Based on the total computation time, the residual latency, and the fixed overhead, a training step time consumption model containing multiple training parameters is constructed.

[0017] Optionally, determining the search space based on multiple sets of candidate training configurations includes:

[0018] Based on each group of candidate training configurations, extract the value set of each training parameter, with each training parameter corresponding to a value set;

[0019] The set of values ​​for the training parameters is used to determine the search space for each training parameter, with each training parameter corresponding to one search space.

[0020] Optionally, the step of jointly optimizing multiple training parameters under preset constraints through structured grid search to find the configuration combination that maximizes effective throughput, thereby obtaining the optimal training configuration for the training task, includes:

[0021] Under preset constraints, sparse sampling processing is performed on multiple training parameters through structured grid search to obtain several representative training configurations;

[0022] Calculate the effective throughput corresponding to several of the representative training configurations;

[0023] Based on the effective throughput, a cryptographic search is performed near the representative training configuration with the highest effective throughput to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0024] Optionally, the step of entering the training phase and hot-tuning the training parameters by tracking changes in real-time communication masking rate includes:

[0025] During the training phase, the current communication masking rate is obtained by periodically sampling the communication masking rate in real time.

[0026] Based on the current communication masking rate, the exponential moving average of the current communication masking rate is calculated;

[0027] When the deviation of the current communication masking rate from the exponential moving average exceeds a preset deviation threshold, the training parameters are hot-tuned.

[0028] Secondly, embodiments of this application provide a deep learning training apparatus based on communication masking rate, the deep learning training apparatus based on communication masking rate comprising:

[0029] The acquisition module is used to acquire multiple sets of candidate training configurations during the configuration phase in the first K training steps after the training task starts. For each set of candidate training configurations, it collects the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side. Based on the computation time, communication time, and combined training time, it calculates the communication masking rate corresponding to each set of candidate training configurations. The candidate training configurations are used to configure multiple training parameters required by the training task. The communication masking rate is used to quantify the degree of overlap between computation and communication in the training task. The computation time is collected by disabling AllReduce callbacks on the pure computation side. The communication time is collected by skipping computation on the pure communication side and only executing AllReduce callbacks on tensors of the same size as the gradient. The combined training time is collected by normal training on the computation and communication side.

[0030] A construction module is used to construct a training step time model containing multiple training parameters based on the communication masking rate;

[0031] The joint optimization module is used to determine the search space based on multiple sets of candidate training configurations, and under preset constraints, to jointly optimize multiple training parameters through structured grid search to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0032] The configuration module is used to configure training parameters for the training task using the optimal training configuration, complete the configuration phase, enter the training phase, and perform hot adjustment of the training parameters by tracking changes in real-time communication masking rate.

[0033] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the deep learning training method based on communication masking rate provided in embodiments of the present invention.

[0034] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps in the deep learning training method based on communication masking rate provided in the embodiments of the present invention.

[0035] The above-mentioned scheme of this application has the following beneficial effects: After the training task starts, the configuration phase is entered in the first K training steps. Multiple sets of candidate training configurations and the communication masking rate corresponding to each set of candidate training configurations are obtained. The candidate training configurations are used to configure multiple training parameters required by the training task, and the communication masking rate is used to quantify the degree of overlap between the computation and communication of the training task. Based on the communication masking rate, a training step time model containing multiple training parameters is constructed. The search space is determined according to the multiple sets of candidate training configurations, and under preset constraints, multiple training parameters are jointly optimized through structured grid search to find the configuration combination that maximizes the effective throughput, thus obtaining the optimal training configuration for the training task. The optimal training configuration is used to configure the training parameters for the training task, completing the configuration phase and entering the training phase. By tracking the changes in the real-time communication masking rate, the training parameters are hot-tuned. This invention solves the problems of existing methods where the performance model either assumes complete overlap or complete non-overlap, both of which deviate significantly from reality. Parameter tuning is usually limited to a single dimension, and there is no joint optimization of local batch size, gradient accumulation steps, and communication bucket size, lacking the ability to adapt to changes in the runtime environment.

[0036] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating a deep learning training method based on communication masking rate, provided as an embodiment of this application;

[0039] Figure 2 This is a schematic diagram of the structure of a deep learning training device based on communication masking rate according to an embodiment of this application;

[0040] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0042] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0043] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0044] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0045] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0046] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0047] like Figure 1 As shown, Figure 1 This is a flowchart of a deep learning training method based on communication masking rate provided in an embodiment of the present invention. The deep learning training method based on communication masking rate includes the following steps:

[0048] 101. After the training task starts, the configuration phase is entered in the first K training steps. Multiple sets of candidate training configurations are obtained. For each set of candidate training configurations, the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side are collected. Based on the computation time, communication time, and combined training time, the communication masking rate corresponding to each set of candidate training configurations is calculated.

[0049] In this embodiment of the invention, the deep learning training method based on communication masking rate can be applied to a deep learning training platform. This platform can be built on a server-based or distributed system and includes a data interface (for sensor or user uploads), a knowledge database, and a knowledge database construction program. The data interface can be used to enter the configuration phase during the first K training steps after the training task starts, obtaining multiple sets of candidate training configurations and the corresponding communication masking rate for each set. The knowledge database construction program can be used to construct the knowledge database, which is specifically used to provide additional association information for the identified data entities, improving the depth of the data recognition system's understanding of the content.

[0050] The training task mentioned above can be a complete distributed deep learning training job, including model initialization, data loading, iterative optimization of multiple training steps, and the complete distributed training process until the model converges or reaches the stopping condition.

[0051] The aforementioned first K training steps can be the first K training steps after the start of distributed training, used for performance probing and configuration optimization. These training steps can be standard training iterations in distributed deep learning training, including forward propagation, backpropagation, gradient synchronization, and parameter updates. The first K steps can be from step 1 to step K, excluding steps after K. K is a positive integer, such as 6, 8, 12, etc.

[0052] The configuration phase described above can be the period from the start of the training task to the period before formal training.

[0053] The above candidate training configurations are used to configure multiple training parameters required for the training task. These training parameters include the local batch size B, gradient accumulation steps G, and communication bucket size S.

[0054] The communication masking ratio described above is used to quantify the degree of overlap between computation and communication in the training task. The communication masking ratio (μ) ranges from [0, 1]. μ=1 indicates that communication is completely masked by computation, and μ=0 indicates that communication and computation are completely serial. The communication masking ratio is a hardware-dependent, load-dependent parameter that changes dynamically with runtime state.

[0055] The computation time mentioned above was obtained by disabling AllReduce callbacks on the pure computation side. This computation time could be the time consumed by performing only forward and backward propagation under conditions where communication interference is completely eliminated.

[0056] The pure computation side described above can be a runtime mode used to calculate pure computation time. On the pure computation side, AllReduce is disabled, and only forward and backward propagation are performed, thus eliminating any interference from communication on computation time. On the pure computation side, no-operations can be inserted into the AllReduce callback to preserve the complete execution path of the computation side. AllReduce, as mentioned above, is a collective communication primitive in a distributed system. AllReduce is used to aggregate the gradients calculated by all acceleration cores in distributed training and distribute the aggregated results to each core, thereby ensuring that the model parameters on each core remain consistent.

[0057] The aforementioned communication time was obtained by skipping computation on the pure communication side and only executing the AllReduce callback on a tensor of the same size as the gradient. This communication time could also be the time consumed by executing a complete AllReduce callback once on a random tensor of the same size as the model gradient without performing any computation.

[0058] The aforementioned pure communication side can be an operating mode used to measure pure communication time. On the pure communication side, the training task skips all computations and only performs the AllReduce callback on a tensor of the same size as the gradient. The pure communication side eliminates computational interference and reflects the baseline performance of the communication subsystem.

[0059] The aforementioned comprehensive training time is obtained through normal training on the computation and communication combined side. This comprehensive training time can be the total end-to-end time required to complete a logic training operation through normal training on the computation and communication combined side. The aforementioned computation and communication combined side can be an operating mode used to measure the total end-to-end time when computation and communication are executed in parallel.

[0060] It should be noted that the introduction of communication masking transforms the previously overlooked dynamic overlap between computation and communication into a quantifiable and modelable explicit parameter, providing a reliable data foundation for subsequent implementation. For each candidate training configuration, the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side can be collected R times. The median of the R times is then taken to eliminate occasional fluctuations. Here, R is a positive integer, such as 3, 5, 6, etc., with a typical value of R=3.

[0061] 102. Based on the communication masking rate, construct a training step time model containing multiple training parameters.

[0062] In this embodiment of the invention, the communication masking rate reflects what proportion of communication time is calculated to be masked under the current hardware and current candidate training configuration.

[0063] The training step time model mentioned above can be a mathematical expression that includes multiple training parameters and is constructed based on the communication masking rate, used to predict or estimate the end-to-end time required for a single training step in distributed deep learning training.

[0064] The training step time model described above can be used to estimate the end-to-end time required to complete one parameter update given the training parameters.

[0065] 103. Determine the search space based on multiple candidate training configurations, and under preset constraints, jointly optimize multiple training parameters through structured grid search to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0066] In this embodiment of the invention, the above-described candidate training configuration is used to configure multiple training parameters required for the training task. The training parameters include the local batch size B, the gradient accumulation steps G, and the communication bucket size S. Candidate values ​​for the local batch size B can be {1, 2, 4, 8, ..., B_max}, candidate values ​​for the gradient accumulation steps G can be {1, 2, 4, 8}, and candidate values ​​for the communication bucket size S can be {5MB, 10MB, 25MB, 50MB, 100MB}.

[0067] The search space described above can be a set consisting of multiple candidate training configurations. The search space can be a discrete, multidimensional set of parameter values, which can be represented as a subset of the local batch size B candidate set × gradient accumulation steps G candidate set × communication bucket size S candidate set.

[0068] In one possible implementation, for example, if the candidate values ​​for the local batch size B in the candidate training configuration are {4,8,16,32}, the candidate values ​​for the gradient accumulation steps G are {1,2,4,8}, and the candidate values ​​for the communication bucket size S are {25MB,50MB}, then the search space = {4,8,16,32} × {1,2,4,8} × {25 MB,50MB}.

[0069] The aforementioned preset constraints can be pre-set hard conditions or limitations that must be met. These preset constraints ensure that the searched training configuration does not exceed hardware capabilities during actual runtime, nor do they impair model convergence. Preset constraints include memory constraints and global batch size constraints. For example, under memory constraints, candidate values ​​for the local batch size B can be {1, 2, 4, 8, ..., B_max}, where B ≤ B_max, and B_max is the maximum local batch size supported by a single acceleration core. Under global batch size constraints, candidate values ​​for the gradient accumulation steps G can be {1, 2, 4, 8}; and candidate values ​​for the communication bucket size S can be {5MB, 10MB, 25MB, 50MB, 100MB}, covering typical value ranges for small, medium, and large buckets. The aforementioned memory constraint can also mean that the local batch size B cannot exceed the maximum batch size that the memory capacity of a single acceleration core can accommodate. The global batch size constraint described above limits the upper limit of the total number of training samples processed by all acceleration cores in a single logical training step. The mathematical form of the global batch size constraint can be expressed as:

[0070] B × G × N ≤ GlobalBatch_max

[0071] Where B represents the local batch size; G represents the gradient accumulation steps; N represents the total number of acceleration cores, such as the number of GPUs; and GlobalBatch_max is the preset upper limit of the global batch size. It's understandable that the global batch size has a significant impact on the model's convergence and generalization ability. An excessively large global batch size can lead to training instability, slower convergence, and decreased generalization performance, especially when using large batches for training, requiring adjustments to the learning rate or the use of special optimization techniques. Conversely, an excessively small global batch size can prevent full utilization of hardware parallelism and result in significant gradient noise. The global batch size constraint is designed to prevent the search algorithm from selecting configurations beyond a reasonable range, thus balancing training efficiency and convergence quality.

[0072] The structured grid search described above can be a method or strategy that, within a search space and under predefined constraints, jointly optimizes multiple training parameters to find the optimal configuration combination that maximizes effective throughput. Alternatively, it can be a method that exhaustively searches all representative training configurations among predefined training parameters to find the optimal model parameters. Structured grid search can employ a phased approach, starting with a coarse search and then refining the search, significantly reducing the number of configurations that need to be actually explored.

[0073] The aforementioned joint optimization can be a process of determining the search space based on multiple sets of candidate training configurations, and then, under preset constraints, jointly optimizing multiple training parameters through structured grid search to find the configuration combination that maximizes the effective throughput.

[0074] The aforementioned effective throughput is a core performance metric used to measure the efficiency of distributed deep learning training. It is used to quantify the number of effective samples that a distributed training system can process per unit of time.

[0075] Specifically, the effective throughput Θ can be expressed as:

[0076] Θ(B, G, S) = (N × B × G) / T_step(B, G, S)

[0077] Where Θ is the effective throughput, B is the local batch size, G is the gradient accumulation step, S is the communication bucket size, N is the total number of acceleration cores, and T_step is the training step time of the model.

[0078] Understandably, the joint optimization objective is to maximize the effective throughput under the constraints of memory constraint B ≤ B_max and global batch size constraint B × G × N ≤ GlobalBatch_max.

[0079] The optimal training configuration mentioned above can be determined by defining a search space based on multiple candidate training configurations and, under preset constraints, jointly optimizing multiple training parameters through structured grid search to find the configuration combination that maximizes effective throughput. The optimal training configuration includes the optimal local batch size B, the optimal gradient accumulation steps G, and the optimal communication bucket size S. The optimal training configuration maximizes effective throughput while satisfying the memory constraint B ≤ B_max and the global batch size constraint B×G×N ≤ GlobalBatch_max.

[0080] 104. Configure training parameters for the training task using the optimal training configuration, complete the configuration phase, and enter the training phase. Perform hot adjustment of training parameters by tracking changes in real-time communication masking rate.

[0081] In this embodiment of the invention, the above-mentioned training parameter configuration may be to set the optimal training configuration into the training task so that the training task is executed according to the optimal training configuration.

[0082] The training phase described above can be the phase after the configuration phase, where formal and continuous model convergence training is performed using the optimal training configuration.

[0083] The aforementioned tracking of changes in real-time communication masking rate can be performed during the training phase, continuously tracking the trend of communication masking rate changes in a low-overhead manner.

[0084] The aforementioned hot-tuning can be a process of dynamically modifying training parameters based on changes in communication masking rate without stopping the training task. During hot-tuning, instead of re-executing the complete structured grid search, a dense search is performed near the optimal training configuration to quickly converge to the new optimal training configuration.

[0085] In this embodiment of the invention, a training step time model containing multiple training parameters is constructed using the communication masking rate corresponding to each set of candidate training configurations, avoiding the two coarse assumptions of complete overlap or complete non-overlap. The search space is determined based on multiple sets of candidate training configurations, and under preset constraints, multiple training parameters are jointly optimized through structured grid search. Joint optimization can discover better configuration combinations that cannot be achieved through single-dimensional parameter tuning. The optimal training configuration is used to configure training parameters for the training task, completing the configuration phase and entering the training phase. By tracking real-time changes in the communication masking rate, the training parameters are hot-tuned. After training enters the formal phase, the communication masking rate is continuously tracked in a low-overhead manner, enabling the system to perceive runtime factors such as bus load changes, hardware frequency reduction, and resource contention among other processes, and automatically adjust the configuration. This solves the problems of existing methods where the performance model either assumes complete overlap or complete non-overlap, both of which deviate significantly from reality. Parameter tuning is usually limited to a single dimension, failing to jointly optimize local batch size, gradient accumulation steps, and communication bucket size, and lacking adaptability to changes in the runtime environment.

[0086] In this embodiment of the invention, after the training task starts, the configuration phase is entered in the first K training steps. Multiple candidate training configurations and the corresponding communication masking rate for each candidate configuration are obtained. The candidate training configurations are used to configure multiple training parameters required by the training task, and the communication masking rate is used to quantify the degree of overlap between computation and communication in the training task. Based on the communication masking rate, a training step time model containing multiple training parameters is constructed. The search space is determined according to the multiple candidate training configurations, and under preset constraints, multiple training parameters are jointly optimized through structured grid search to find the configuration combination that maximizes effective throughput, thus obtaining the optimal training configuration for the training task. The optimal training configuration is used to configure the training parameters for the training task, completing the configuration phase and entering the training phase. The training parameters are hot-tuned by tracking changes in the real-time communication masking rate. This invention solves the problems of existing methods where the performance model either assumes complete overlap or complete non-overlap, both of which deviate significantly from reality. Parameter tuning is usually limited to a single dimension, and there is no joint optimization of local batch size, gradient accumulation steps, and communication bucket size, lacking adaptability to changes in the runtime environment.

[0087] It is understood that in the specific implementation of this application, data such as training data, configuration data, and parameter data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0088] Optionally, in the step of calculating the communication masking rate corresponding to each group of candidate training configurations based on the computation time, communication time, and overall training time, the additional latency actually exposed by communication can be calculated based on the computation time and overall training time; and the communication masking rate corresponding to each group of candidate training configurations can be calculated based on the additional latency and communication time.

[0089] In this embodiment of the invention, the additional latency actually exposed by the communication may be that during normal training in which computation and communication are parallel, the communication operation is not masked by the computation process, thereby adding extra time to the total training step.

[0090] Specifically, the communication masking rate can be expressed as:

[0091] μ = 1 − (T_both − T_comp) / T_comm

[0092] Where μ is the communication masking rate, T_both is the training time, T_comp is the computation time, T_comm is the communication time, and T_both − T_comp is the additional latency actually exposed by communication, i.e., the communication time not masked by computation. Dividing the additional latency actually exposed by communication by the communication time T_comm gives the unmasked proportion. Subtracting the unmasked proportion from 1 gives the communication masking rate μ. When μ = 1, T_both = T_comp, and communication is completely masked by computation; when μ = 0, T_both = T_comp + T_comm, and computation and communication are completely serial.

[0093] The aforementioned communication masking rate is used to quantify the degree of overlap between computation and communication in the training task.

[0094] It should be noted that the communication masking rate is not only an inherent property of the hardware, but also related to the current training configuration. A larger local batch size B means longer computation time and easier masking of communications; a larger communication bucket size S means that communications are more concentrated and bursty, and more difficult to mask. Therefore, the communication masking rate is a function of the training parameters μ(B,G,S).

[0095] Optionally, in the step of constructing a training step time-consuming model containing multiple training parameters based on the communication masking rate, the total computation time of step gradient accumulation, the residual delay of unmasked communication, and the fixed cost of parameter update can be calculated based on the communication masking rate; and the training step time-consuming model containing multiple training parameters can be constructed based on the total computation time, residual delay, and fixed cost.

[0096] In this embodiment of the invention, the total computation time for gradient accumulation can be the total computation time calculated based on the number of gradient accumulation steps. The total computation time can be the total computation time spent completing G forward and backward propagation steps in gradient accumulation mode, where G is the number of gradient accumulation steps.

[0097] The unmasked residual delay in the aforementioned communication may be due to the communication operation not being masked by the computation process during training where computation and communication are parallel, thus adding extra communication time to the training step duration.

[0098] The fixed overhead of the parameter update can be the time spent performing the fixed parameter update operation. The fixed overhead of the parameter update does not change with the local batch size B, gradient accumulation steps G, or bucket size S. The fixed overhead of the parameter update includes optimizer step size, learning rate scheduling, etc.

[0099] The training step time consumption model mentioned above can be a mathematical expression that includes multiple training parameters, constructed based on total computation time, residual latency, and fixed overhead, and is used to predict or estimate the end-to-end time required for a single training step in distributed deep learning training.

[0100] Specifically, the training step time model can be represented as:

[0101] T_step(B, G, S) = G × T_comp(B) + (1 − μ(B, G, S)) × T_comm(S) + T_update

[0102] Where T_step(B, G, S) is the training step time model; G is the gradient accumulation step number; T_comp(B) is the computation time, which is obtained by disabling the AllReduce callback in the pure computation side; G × T_comp(B) is the total computation time for gradient accumulation in one step; μ(B, G, S) is the communication masking rate; T_comm(S) is the communication time, which is obtained by skipping computation in the pure communication side and only performing the AllReduce callback on tensors of the same size as the gradient; (1 −μ(B, G, S)) × T_comm(S) is the residual delay that is not masked in the communication; and T_update is the fixed cost of parameter updates.

[0103] Optionally, in the step of determining the search space based on multiple sets of candidate training configurations, the set of values ​​for each training parameter can be extracted based on each set of candidate training configurations, with each training parameter corresponding to a set of values; the set of values ​​for the training parameters can be used to determine the search space for each training parameter, with each training parameter corresponding to a search space.

[0104] In this embodiment of the invention, the above-mentioned candidate training configuration is used to configure multiple training parameters required for the training task. The training parameters include the local batch size B, the gradient accumulation steps G, and the communication bucket size S.

[0105] The set of values ​​for the training parameters mentioned above can be a set of specific numerical values ​​for the training parameters. For example, the set of values ​​for the local batch size B can be {1,2,4,8,..., B_max}, the set of values ​​for the gradient accumulation steps G can be {1,2,4,8}, and the set of values ​​for the communication bucket size S can be {5MB,10MB,25MB,50MB,100MB}, etc.

[0106] The search space described above can be the set of each training parameter determined based on the set of values ​​for the training parameters. The search space can be a discrete, multidimensional set of parameter values, which can be represented as a subset of the set of values ​​for the local batch size B × the set of values ​​for the gradient accumulation steps G × the set of values ​​for the communication bucket size S.

[0107] In one possible implementation, for example, if the set of values ​​for the local batch size B in the candidate training configuration is {4,8,16,32}, the set of values ​​for the gradient accumulation steps G is {1,2,4,8}, and the set of values ​​for the communication bucket size S is {25MB,50MB}, then the search space = {4,8,16,32} × {1,2,4,8} × {25MB,50MB}.

[0108] It should be noted that each training parameter corresponds to a set of values, and each training parameter corresponds to a search space.

[0109] Optionally, in the step of jointly optimizing multiple training parameters through structured grid search to find the configuration combination that maximizes the effective throughput and obtain the optimal training configuration for the training task, under preset constraints, multiple training parameters can be sparsely sampled through structured grid search to obtain several representative training configurations; the effective throughput corresponding to several representative training configurations can be calculated; based on the effective throughput, a dense search can be performed near the representative training configuration with the highest effective throughput to find the configuration combination that maximizes the effective throughput and obtain the optimal training configuration for the training task.

[0110] In this embodiment of the invention, the aforementioned preset constraints can be pre-set constraints. These constraints ensure that the searched training configuration does not exceed hardware capabilities during actual runtime, nor does it impair model convergence. Constraints include memory constraints and global batch size constraints. Under memory constraints, the set of values ​​for the local batch size B is {1, 2, 4, 8, ..., B_max}, where B ≤ B_max, and B_max is the maximum local batch size supported by a single acceleration core. Under global batch size constraints, the set of values ​​for the gradient accumulation steps G is {1, 2, 4, 8}; and the set of values ​​for the communication bucket size S is {5MB, 10MB, 25MB, 50MB, 100MB}, covering typical value ranges for small, medium, and large buckets. The aforementioned memory constraints can mean that the local batch size B cannot exceed the maximum batch size that the memory capacity of a single acceleration core can accommodate. The aforementioned core can be a GPU, etc. The global batch size constraint described above limits the upper limit of the total number of training samples processed by all acceleration cores in a single logical training step. The mathematical form of the global batch size constraint can be expressed as:

[0111] B × G × N ≤ GlobalBatch_max

[0112] Where B represents the local batch size; G represents the gradient accumulation steps; N represents the total number of acceleration cores, such as the number of GPUs; and GlobalBatch_max is the preset upper limit of the global batch size. It's understandable that the global batch size has a significant impact on the model's convergence and generalization ability. An excessively large global batch size can lead to training instability, slower convergence, and decreased generalization performance, especially when using large batches for training, requiring adjustments to the learning rate or the use of special optimization techniques. Conversely, an excessively small global batch size can prevent full utilization of hardware parallelism and result in significant gradient noise. The global batch size constraint is designed to prevent the search algorithm from selecting configurations beyond a reasonable range, thus balancing training efficiency and convergence quality.

[0113] The aforementioned structured grid search can be a method or strategy that, under preset constraints, jointly optimizes multiple training parameters to find the optimal configuration combination that maximizes effective throughput.

[0114] The structured grid search described above can be a method that exhaustively searches for all representative training configurations among predefined training parameters to find the optimal model parameters. Structured grid search can employ a phased approach of sparse sampling followed by intensified searching, significantly reducing the number of configurations that need to be actually explored.

[0115] The sparse sampling process described above can be a process of sparsely sampling multiple training parameters through structured grid search under preset constraints to select several representative training configurations. The purpose of sparse sampling is to quickly cover the general trend of the entire search space and find several representative training configurations with a small number of probes.

[0116] The aforementioned representative training configuration can be a representative training configuration selected by sparsely sampling multiple training parameters through structured grid search under constraints.

[0117] Furthermore, for each representative training configuration, the communication masking rate and effective throughput corresponding to that configuration are calculated. Specifically, for each representative training configuration, based on the communication masking rate measured for that configuration, the effective throughput under that configuration is calculated using a training step time model.

[0118] The aforementioned effective throughput is a core performance metric used to measure the efficiency of distributed deep learning training. It is used to quantify the number of effective samples that a distributed training system can process per unit of time.

[0119] Specifically, the effective throughput Θ can be expressed as:

[0120] Θ(B, G, S) = (N × B × G) / T_step(B, G, S)

[0121] Where Θ is the effective throughput, B is the local batch size, G is the gradient accumulation step, S is the communication bucket size, N is the total number of acceleration cores, and T_step is the training step time of the model.

[0122] The aforementioned encrypted search can be a process of performing an encrypted search around the representative configuration point with the highest effective throughput to find the configuration combination that maximizes the effective throughput. Encrypted search can compensate for local optima that sparse sampling may miss due to excessively large step sizes. By reducing the step size, it can accurately locate the optimal training configuration without causing a surge in the total number of probes.

[0123] The optimal training configuration mentioned above can be found by performing an encrypted search around the representative training configuration with the highest effective throughput, based on the effective throughput, to find the combination of configurations that maximizes the effective throughput.

[0124] It should be noted that the joint optimization objective is to maximize the effective throughput under the constraints of memory constraint B ≤ B_max and global batch size constraint B × G × N ≤ GlobalBatch_max.

[0125] Optionally, in the step of hot-adjusting training parameters by tracking changes in real-time communication masking rate during the training phase, the current communication masking rate is obtained by periodically sampling the communication masking rate in real time; based on the current communication masking rate, the exponential moving average of the current communication masking rate is calculated; when the deviation of the current communication masking rate from the exponential moving average exceeds a preset deviation threshold, the training parameters are hot-adjusted.

[0126] In this embodiment of the invention, after entering the training phase, the communication masking rate can be continuously tracked in a low-overhead manner, and the communication masking rate can be periodically collected to obtain the current communication masking rate.

[0127] The aforementioned periodic sampling can be a process of sampling the masking rate at a sampling period. The sampling period can be every M steps to sample the communication masking rate, where M can be 300, 400, 500, 600, etc., and a typical value of M can be 500.

[0128] The aforementioned current communication masking rate can be obtained by periodically sampling the communication masking rate in real time during the training phase. The current communication masking rate reflects the degree of overlap between computation and communication at the current moment.

[0129] The aforementioned exponential moving average can be the current exponential moving average of the communication masking rate, or it can be obtained by exponentially weighting and smoothing the historically sampled communication masking rate. The exponential moving average can be denoted as μ_ema. The exponential moving average is used to filter out short-term random fluctuations and stably reflect the changing trend of the communication masking rate. It is a key benchmark for determining whether a significant drift has occurred, thereby triggering a configuration hot adjustment.

[0130] The aforementioned preset deviation threshold can be a pre-set deviation threshold used to determine whether the deviation of the current communication masking rate from its exponential moving average has reached a significant level. The preset deviation threshold can be 0.15, 0.18, 0.2, etc.

[0131] The aforementioned hot-tuning can be a process of dynamically and online modifying training parameters during the training phase to adapt to changes in the current environment. During hot-tuning, instead of re-executing the complete structured grid search, a dense search is performed near the optimal training configuration to quickly converge to the new optimal training configuration. This dense search can involve adjusting only levels B±1, G±1, and S±1. Adjusting only levels B±1, G±1, and S±1 can be understood as performing a dense search near the optimal training configuration, which allows the training parameters in the optimal training configuration to be changed upwards or downwards by a discrete candidate step size based on their current values. For example, the set of values ​​for the local batch size B is {4, 8, 16, 32}, the set of values ​​for the gradient accumulation steps G is {1, 2, 4, 8}, and the set of values ​​for the communication bucket size S is {25 MB, 50 MB, 100 MB}. The optimal training configuration is B=16, G=4, S=50MB. When performing encrypted search near the optimal training configuration, the local batch size B can take the value 8 before 16 and the value 32 after 16, resulting in the set of values ​​{8, 16, 32}; the gradient accumulation steps G can take the value 2 before 4 and the value 8 after 4, resulting in the set of values ​​{2, 4, 8}; and the communication bucket size S can take the value 25 before 50 MB and the value 100 MB after 50 MB, resulting in the set of values ​​{25MB, 50MB, 100MB}.

[0132] In one possible implementation, for example, when the preset deviation threshold is 0.15, the exponential moving average of the current communication masking rate μ_current is μ_ema. When the deviation of the current communication masking rate from the exponential moving average |μ_current − μ_ema| / μ_ema exceeds 0.15, it is determined that the current communication masking rate has changed significantly, and configuration re-optimization is triggered to hot-adjust the training parameters.

[0133] In this embodiment of the invention, after entering the training phase, changes in the communication masking rate can be continuously tracked in a low-overhead manner. The communication masking rate is periodically collected, and an exponential moving average of the communication masking rate is maintained. When the deviation of the current communication masking rate from the exponential moving average exceeds a preset deviation threshold, the training parameters are hot-adjusted. Online tracking of the communication masking rate enables the system to sense runtime factors such as changes in bus load, hardware frequency reduction, and resource contention of other processes, and to hot-adjust the training parameters accordingly.

[0134] It should be noted that after hot-tuning the training parameters, the learning rate is updated synchronously to maintain a linear scaling relationship between the global batch size and the learning rate. Understandably, hot-tuning changes the training parameters, causing a change in the global batch size. The learning rate of the model's training step time is then updated synchronously according to a linear scaling rule to maintain training stability and convergence.

[0135] like Figure 2 As shown, this embodiment of the invention provides a deep learning training device based on communication masking rate, which includes:

[0136] The acquisition module 201 is used to acquire multiple sets of candidate training configurations during the configuration phase in the first K training steps after the training task starts. For each set of candidate training configurations, the module collects the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side. Based on the computation time, communication time, and combined training time, the module calculates the communication masking rate corresponding to each set of candidate training configurations. The candidate training configurations are used to configure multiple training parameters required by the training task. The communication masking rate is used to quantify the degree of overlap between computation and communication in the training task. The computation time is collected by disabling the AllReduce callback on the pure computation side. The communication time is collected by skipping computation on the pure communication side and only executing the AllReduce callback on tensors of the same size as the gradient. The combined training time is collected by normal training on the computation and communication side.

[0137] The construction module 202 is used to construct a training step time model containing multiple training parameters based on the communication masking rate;

[0138] The joint optimization module 203 is used to determine the search space based on multiple sets of candidate training configurations, and under preset constraints, to jointly optimize multiple training parameters through structured grid search to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0139] The configuration module 204 is used to configure training parameters for the training task through the optimal training configuration, complete the configuration phase, enter the training phase, and perform hot adjustment of the training parameters by tracking the changes in real-time communication masking rate.

[0140] Optionally, the acquisition module 201 is further configured to calculate the additional latency actually exposed by communication based on the calculation time and the comprehensive training time; and to calculate the communication masking rate corresponding to each group of candidate training configurations based on the additional latency and the communication time.

[0141] Optionally, the construction module 202 is further configured to calculate the total computation time of step gradient accumulation, the residual delay that is not masked in the communication, and the fixed cost of parameter update based on the communication masking rate; and to construct a training step time consumption model containing multiple training parameters based on the total computation time, the residual delay, and the fixed cost.

[0142] Optionally, the joint optimization module 203 is further configured to extract the value set of each training parameter based on each group of candidate training configurations, with each training parameter corresponding to a value set; and to determine the search space of each training parameter from the value set of the training parameters, with each training parameter corresponding to a search space.

[0143] Optionally, the joint optimization module 203 is further configured to perform sparse sampling processing on multiple training parameters through structured grid search under preset constraints to obtain several representative training configurations; calculate the effective throughput corresponding to several representative training configurations; and based on the effective throughput, perform a dense search near the representative training configuration with the highest effective throughput to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0144] Optionally, the configuration module 204 is further configured to enter the training phase, periodically sample the communication masking rate in real time to obtain the current communication masking rate; calculate the exponential moving average of the current communication masking rate based on the current communication masking rate; and perform hot adjustment of the training parameters when the deviation of the current communication masking rate from the exponential moving average exceeds a preset deviation threshold.

[0145] like Figure 3 As shown, embodiments of the present invention also provide an electronic device, including a processor, which can execute any of the above-described deep learning training methods based on communication masking rate.

[0146] Specifically, it includes a processor 301 and a memory 302, as well as a computer program stored in the memory 302 and capable of running on the processor 301, which executes a deep learning training method based on communication masking rate, wherein:

[0147] The processor 301 executes the calculator program based on the deep learning training method with communication masking rate stored in the memory 302, and performs the following steps:

[0148] After the training task starts, the configuration phase is entered in the first K training steps. Multiple sets of candidate training configurations are obtained, and for each set of candidate training configurations, the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side are collected. Based on the computation time, communication time, and combined training time, the communication masking rate corresponding to each set of candidate training configurations is calculated. The candidate training configuration is used to configure multiple training parameters required by the training task. The communication masking rate is used to quantify the degree of overlap between computation and communication in the training task. The computation time is collected by disabling AllReduce callbacks on the pure computation side. The communication time is collected by skipping computation on the pure communication side and only executing AllReduce callbacks on tensors of the same size as the gradient. The combined training time is collected by normal training on the computation and communication side.

[0149] Based on the communication masking rate, a training step time model containing multiple training parameters is constructed;

[0150] The search space is determined based on multiple sets of candidate training configurations, and under preset constraints, multiple training parameters are jointly optimized through structured grid search to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0151] The optimal training configuration is used to configure the training parameters for the training task, completing the configuration phase and entering the training phase. The training parameters are then hot-tuned by tracking changes in the real-time communication masking rate.

[0152] Optionally, the processor 301 performs the step of calculating the communication masking rate corresponding to each group of candidate training configurations based on the computation time, communication time, and overall training time, including:

[0153] Based on the computation time and the comprehensive training time, the additional latency actually exposed by the communication is calculated.

[0154] Based on the additional latency and the communication time, the communication masking rate corresponding to each group of candidate training configurations is calculated.

[0155] Optionally, the process executed by processor 301 to construct a training step time-consuming model containing multiple training parameters based on the communication masking rate includes:

[0156] Based on the communication masking rate, calculate the total computation time for step gradient accumulation, the residual delay of unmasked communication, and the fixed overhead of parameter updates;

[0157] Based on the total computation time, the residual latency, and the fixed overhead, a training step time consumption model containing multiple training parameters is constructed.

[0158] Optionally, the process of determining the search space based on multiple sets of candidate training configurations, performed by processor 301, includes:

[0159] Based on each group of candidate training configurations, extract the value set of each training parameter, with each training parameter corresponding to a value set;

[0160] The set of values ​​for the training parameters is used to determine the search space for each training parameter, with each training parameter corresponding to one search space.

[0161] Optionally, the processor 301 performs the following step: under preset constraints, it performs joint optimization of multiple training parameters through structured grid search to find the configuration combination that maximizes effective throughput, thereby obtaining the optimal training configuration for the training task. This includes:

[0162] Under preset constraints, sparse sampling processing is performed on multiple training parameters through structured grid search to obtain several representative training configurations;

[0163] Calculate the effective throughput corresponding to several of the representative training configurations;

[0164] Based on the effective throughput, a cryptographic search is performed near the representative training configuration with the highest effective throughput to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

[0165] Optionally, the processor 301 executes the entry into the training phase, and performs hot-tuning of the training parameters by tracking changes in real-time communication masking rate, including:

[0166] During the training phase, the current communication masking rate is obtained by periodically sampling the communication masking rate in real time.

[0167] Based on the current communication masking rate, the exponential moving average of the current communication masking rate is calculated;

[0168] When the deviation of the current communication masking rate from the exponential moving average exceeds a preset deviation threshold, the training parameters are hot-tuned.

[0169] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the deep learning training method based on communication masking rate provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0170] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A deep learning training method based on communication masking rate, characterized in that, The method includes the following steps: After the training task starts, the configuration phase is entered in the first K training steps. Multiple sets of candidate training configurations are obtained, and for each set of candidate training configurations, the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side are collected. Based on the computation time, communication time, and combined training time, the communication masking rate corresponding to each set of candidate training configurations is calculated. The candidate training configuration is used to configure multiple training parameters required by the training task. The communication masking rate is used to quantify the degree of overlap between computation and communication in the training task. The computation time is collected by disabling AllReduce callbacks on the pure computation side. The communication time is collected by skipping computation on the pure communication side and only performing AllReduce callbacks on tensors of the same size as the gradient. The combined training time is collected by normal training on the computation and communication side. Based on the communication masking rate, a training step time model containing multiple training parameters is constructed; The search space is determined based on multiple sets of candidate training configurations, and under preset constraints, multiple training parameters are jointly optimized through structured grid search to find the configuration combination that maximizes the effective throughput, thus obtaining the optimal training configuration for the training task; the effective throughput is expressed as: Θ(B, G, S) = (N × B × G) / T_step(B, G, S) Where Θ is the effective throughput, B is the local batch size, G is the gradient accumulation step, S is the communication bucket size, N is the total number of acceleration cores, and T_step is the training step time of the model. The optimal training configuration is used to configure the training parameters for the training task, completing the configuration phase and entering the training phase. The training parameters are then hot-tuned by tracking changes in the real-time communication masking rate. The step of constructing a training step time model containing multiple training parameters based on the communication masking rate includes: The training step time model is constructed based on the total computation time, residual latency, and fixed overhead. The total computation time is the total computation time calculated based on the number of gradient accumulation steps. The residual latency is the additional communication time added to the training step time when the communication operation is not masked by the computation process during training where computation and communication are parallel. The fixed overhead is the time spent performing fixed operations to update parameters. Specifically, the training step time model is expressed as: T_step(B, G, S) = G × T_comp(B) + (1 − μ(B,G, S)) × T_comm(S) + T_update Where T_step(B, G, S) is the training step time model; T_comp(B) is the computation time; G × T_comp(B) is the total computation time for gradient accumulation in each step; μ(B, G, S) is the communication masking rate; T_comm(S) is the communication time; (1 − μ(B,G, S)) × T_comm(S) is the residual delay that is not masked in the communication; and T_update is the fixed cost of parameter update.

2. The deep learning training method based on communication masking rate as described in claim 1, characterized in that, The step of calculating the communication masking rate corresponding to each group of candidate training configurations based on the computation time, communication time, and overall training time includes: Based on the computation time and the comprehensive training time, the additional latency actually exposed by the communication is calculated. Based on the additional latency and the communication time, the communication masking rate corresponding to each group of candidate training configurations is calculated.

3. The deep learning training method based on communication masking rate as described in claim 1, characterized in that, The step of determining the search space based on multiple sets of candidate training configurations includes: Based on each group of candidate training configurations, extract the value set of each training parameter, with each training parameter corresponding to a value set; The set of values ​​for the training parameters is used to determine the search space for each training parameter, with each training parameter corresponding to one search space.

4. The deep learning training method based on communication masking rate as described in claim 3, characterized in that, The step of jointly optimizing multiple training parameters through structured grid search under preset constraints to find the configuration combination that maximizes effective throughput, thereby obtaining the optimal training configuration for the training task, includes: Under preset constraints, sparse sampling processing is performed on multiple training parameters through structured grid search to obtain several representative training configurations; Calculate the effective throughput corresponding to several of the representative training configurations; Based on the effective throughput, a cryptographic search is performed near the representative training configuration with the highest effective throughput to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task.

5. The deep learning training method based on communication masking rate as described in any one of claims 1 to 3, characterized in that, The process of entering the training phase, which involves hot-tuning the training parameters by tracking changes in real-time communication masking rate, includes: During the training phase, the current communication masking rate is obtained by periodically sampling the communication masking rate in real time. Based on the current communication masking rate, the exponential moving average of the current communication masking rate is calculated; When the deviation of the current communication masking rate from the exponential moving average exceeds a preset deviation threshold, the training parameters are hot-tuned.

6. A deep learning training device based on communication masking rate, characterized in that, For implementing the steps in the deep learning training method based on communication masking rate as described in any one of claims 1 to 5, the deep learning training apparatus based on communication masking rate comprises: The acquisition module is used to acquire multiple sets of candidate training configurations during the configuration phase in the first K training steps after the training task starts. For each set of candidate training configurations, it collects the computation time on the pure computation side, the communication time on the pure communication side, and the combined training time on the computation and communication side. Based on the computation time, communication time, and combined training time, it calculates the communication masking rate corresponding to each set of candidate training configurations. The candidate training configurations are used to configure multiple training parameters required by the training task. The communication masking rate is used to quantify the degree of overlap between computation and communication in the training task. The computation time is collected by disabling AllReduce callbacks on the pure computation side. The communication time is collected by skipping computation on the pure communication side and only executing AllReduce callbacks on tensors of the same size as the gradient. The combined training time is collected by normal training on the computation and communication side. A construction module is used to construct a training step time model containing multiple training parameters based on the communication masking rate; The joint optimization module is used to determine the search space based on multiple sets of candidate training configurations, and under preset constraints, to jointly optimize multiple training parameters through structured grid search to find the configuration combination that maximizes the effective throughput, thereby obtaining the optimal training configuration for the training task; the effective throughput is expressed as: Θ(B, G, S) = (N × B × G) / T_step(B, G, S) Where Θ is the effective throughput, B is the local batch size, G is the gradient accumulation step, S is the communication bucket size, N is the total number of acceleration cores, and T_step is the training step time of the model. The configuration module is used to configure training parameters for the training task using the optimal training configuration, complete the configuration phase, enter the training phase, and perform hot adjustment of the training parameters by tracking changes in real-time communication masking rate.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the deep learning training method based on communication masking rate as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the deep learning training method based on communication masking rate as described in any one of claims 1 to 5.

Images