A distributed training communication acceleration method based on similar gradient filtering
Patent Information
- Application Number
- CN202510664429.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-22
- Publication Date
- 2026-11-03
AI Technical Summary
Existing technologies in distributed deep neural network training neglect the gradient relationship between multiple iterations, resulting in insufficient communication optimization and failure to adequately reduce communication volume.
A gradient-based filtering method is adopted, which uses a gradient filtering algorithm to select gradient subsets based on similarity thresholds, updates the aggregation results using a parameter service library, monitors the trend of model loss changes, and dynamically adjusts the gradient similarity thresholds to optimize communication.
It effectively reduces network traffic, improves the communication performance of distributed training, ensures the robustness and acceleration of model training, and protects model quality while minimizing communication volume.
Smart Images
Figure CN123031815A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed training, and in particular to a method for accelerating distributed training communication based on similar gradient filtering. Background Technology
[0002] In recent years, deep neural networks (DNNs) have been widely applied in various fields and have achieved significant breakthroughs. With the continuous increase in model size and training data volume, distributed DNN training has emerged. In distributed training, each worker module trains a portion of the data (or model) separately, and updates the shared model by aggregating the gradients generated by each module through the network. However, the inter-machine communication required for gradient aggregation can lead to network bottlenecks. For larger DNNs (such as GPT-3 with 175 billion parameters) and higher hardware computing speeds (such as faster GPUs like the NVIDIA A100), the communication bottleneck becomes even more pronounced.
[0003] In related technologies, research has explored methods to alleviate communication bottlenecks from different perspectives. However, none of them have noticed the correlation of gradients between multiple iterations during training. Existing work accelerates distributed DNN training at different levels, with most focusing on how to optimize communication. High-performance distributed frameworks or schedulers design efficient parallel strategies or topologies, but these strategies do not consider the characteristics of transmitted gradients; gradient compression and sparse communication schemes reduce gradient transmission through filtering or quantization, but these strategies only consider the gradient characteristics of a single iteration and lack a holistic view of multiple iterations; solutions in network layers (including intra-network aggregation or transport layer optimization) utilize the network's computational or scheduling capabilities to optimize communication, but these strategies assume that all gradients in each iteration will be sent.
[0004] Therefore, the research work provided in the related technologies has limitations. It is a single-iteration communication optimization that ignores the gradient relationship between multiple iterations and thus ignores the space for further optimization of communication. Summary of the Invention
[0005] This application provides a distributed training communication acceleration method based on similar gradient filtering, which at least solves the problem that related techniques are all single-iteration communication optimizations, ignoring the gradient relationship between multiple iterations, and thus neglecting the problem of further optimizing the communication space.
[0006] In a first aspect, this application provides a distributed training communication acceleration method based on similarity gradient filtering, the method comprising:
[0007] Collect the parameter gradients of each working node of the training model;
[0008] The parameter gradient is imported into a preset gradient filtering algorithm, and a gradient subset is obtained by filtering according to the gradient similarity threshold of the gradient filtering algorithm.
[0009] Based on the gradient subset and the pre-configured parameter service library, the aggregation result is updated and returned to the working node;
[0010] Monitor and analyze the loss change trend of the trained model, and adjust the gradient similarity threshold based on the loss change trend.
[0011] Optionally, the step of importing the parameter gradient into a preset gradient filtering algorithm and filtering to obtain a gradient subset based on the gradient similarity threshold of the gradient filtering algorithm includes:
[0012] The working node calculates the similarity of adjacent iteration gradients based on similarity evaluation to obtain the first parameter gradient;
[0013] The gradient of the first parameter is imported into a preset gradient filtering algorithm to obtain the necessary gradient of the second parameter, and the gradient of the second parameter is integrated into a gradient subset.
[0014] Optionally, if the second parameter gradient does not exist in the filter, a first notification is sent to the parameter service library, the first notification indicating that the second parameter gradient does not exist.
[0015] Optionally, updating the aggregated result based on the gradient subset and a pre-configured parameter service library includes:
[0016] A pre-configured parameter service library receives the gradient subsets from different working nodes;
[0017] Based on the header field within the gradient subset, parameter gradient updates are retrieved from the parameter service library to obtain the aggregated result.
[0018] Optionally, monitoring and analyzing the loss change trend of the trained model includes:
[0019] Regularly monitor the training loss value of the distributed training model;
[0020] Calculate the difference in loss value between every two iterations based on the training loss value;
[0021] The trend of loss change in the distributed training model is determined based on the difference in the loss values.
[0022] If the difference is less than or equal to 0, it is considered that the model loss value of the distributed training model has not decreased;
[0023] If the difference is greater than 0 for three consecutive times, the model loss value of the distributed training model is considered to have decreased.
[0024] Optionally, adjusting the gradient similarity threshold based on the loss change trend includes:
[0025] If the difference is less than or equal to 0, the gradient similarity threshold is adjusted to the first gradient similarity threshold to reduce gradient loss.
[0026] If the difference is greater than 0 for three consecutive times, the gradient similarity threshold is dynamically lowered to reduce communication volume. The dynamic lowering formula is as follows:
[0027] S t =S t-1 -adjust_step
[0028] Where S t and S t-1 These are the gradient similarity thresholds before and after adjustment, respectively, and adjust_step is a pre-set fixed adjustment step size.
[0029] Optionally, the first gradient similarity threshold is 0.9.
[0030] Secondly, this application provides a distributed training communication acceleration system based on similarity gradient filtering, the system comprising:
[0031] The worker module includes several worker nodes, which are used to collect the parameter gradients of each worker node in the training model.
[0032] The transmitter module is used to import the parameter gradient into a preset gradient filtering algorithm and filter the gradient subset according to the gradient similarity threshold of the gradient filtering algorithm.
[0033] The remote center module is pre-configured with a parameter service library to record the latest parameter gradient copies of each working node. This allows for querying and retrieving parameter gradients during gradient aggregation. Based on the gradient subset and the parameter service library, the aggregation result is updated and returned to the working node.
[0034] The controller module is used to monitor and analyze the loss change trend of the trained model, and adjust the gradient similarity threshold based on the loss change trend.
[0035] Thirdly, this application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the distributed training communication acceleration method based on similar gradient filtering provided in the first aspect above.
[0036] Compared with related technologies, the distributed training communication acceleration method based on similar gradient filtering provided in this application has at least the following technical advantages:
[0037] By filtering out unnecessary similar gradients through a set transmission strategy without disrupting the aggregation logic, network traffic is effectively reduced. The system's performance and risk are dynamically balanced through the detection of model loss values, and training is accelerated while ensuring model quality by relying on a threshold dynamic adjustment algorithm. This significantly improves the performance of distributed training communication, thereby protecting the robustness of model training while minimizing communication volume.
[0038] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description
[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0040] Figure 1 This is a flowchart illustrating a distributed training communication acceleration method based on similar gradient filtering according to an exemplary embodiment;
[0041] Figure 2 This is an exemplary embodiment illustrating a distributed training communication acceleration system architecture based on similar gradient filtering.
[0042] Figure 3 This is a diagram illustrating a gradient screening algorithm according to an exemplary embodiment;
[0043] Figure 4 This is a performance test diagram comparing the embodiments of this application with existing distributed training communication schemes (NCCL, OmniReduce, PS). Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.
[0045] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.
[0046] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.
[0047] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.
[0048] In related technologies, existing work accelerates distributed DNN training at different levels, with most focusing on how to optimize communication. High-performance distributed frameworks or schedulers design efficient parallel strategies or topologies, but these strategies do not consider the characteristics of the transmitted gradients; gradient compression and sparse communication schemes reduce gradient transmission through filtering or quantization, but these strategies only consider the gradient characteristics of a single iteration and lack a holistic view of multiple iterations; solutions in network layers (including intra-network aggregation or transport layer optimization) utilize the network's computational or scheduling capabilities to optimize communication, but these strategies assume that all gradients in each iteration will be sent.
[0049] Therefore, the aforementioned research works in related technologies all have limitations, as they are all single-iteration communication optimizations that neglect the gradient relationship between multiple iterations, thus ignoring the space for further communication optimization. Based on the above, this invention provides a distributed training communication acceleration method based on similar gradient filtering, which will be described in detail below with reference to specific embodiments and accompanying drawings.
[0050] Example 1
[0051] This invention provides a distributed training communication acceleration method based on similar gradient filtering. Figure 1 This is a flowchart illustrating a distributed training communication acceleration method based on similarity gradient filtering, according to an exemplary embodiment; as follows: Figure 1 As shown, this method basically includes: transmission strategy selection and dynamic adjustment of gradient similarity threshold of gradient screening algorithm;
[0052] Transmission strategy selection:
[0053] Since omitting similar gradients will prevent proper aggregation, and the gradients sent between different worker modules will differ, sparse propagation will inevitably be introduced. Therefore, we designed a novel selective propagation strategy to filter out unnecessary similar gradients without disrupting the aggregation logic. This propagation strategy is as follows:
[0054] S01. Collect the parameter gradients of each working node of the training model;
[0055] S02. The parameter gradient is imported into a preset gradient filtering algorithm, and a gradient subset is obtained by filtering according to the gradient similarity threshold of the gradient filtering algorithm.
[0056] Specifically, the working node calculates the similarity of adjacent iterative gradients based on similarity evaluation to obtain the first parameter gradient; the first parameter gradient is imported into a preset gradient filtering algorithm to filter and obtain the necessary second parameter gradient, and the second parameter gradient is integrated into a gradient subset; if the second parameter gradient is not found in the filter, a first notification is sent to the parameter service library, wherein the first notification is used to indicate that the second parameter gradient is not found.
[0057] Specifically, Figure 3 This is a diagram illustrating a gradient screening algorithm according to an exemplary embodiment; see attached diagram. Figure 3 First, for the gradient of each parameter in the bucket, the worker retrieves the gradient previously sent to the PS using the parameter index. Then, it calculates the similarity score between the two gradients (row 8) and accumulates the similarity error in the corresponding position in the table. If the score is higher than the gradient similarity threshold and the similarity error does not exceed 0.5, the corresponding position in the bitmap is set to 0, indicating that no transmission is needed. Otherwise, the corresponding position in the bitmap is set to 1, indicating that transmission is required. Then, the error field is cleared and the old_gradient field is updated with the current gradient. Finally, the worker obtains a bitmap recording the gradient transmission status.
[0058] S03. Based on the gradient subset and the pre-configured parameter service library, update the obtained aggregation result and return the aggregation result to the working node;
[0059] Specifically, a pre-configured parameter service library receives gradient subsets from different worker nodes; based on the header fields within the gradient subsets, parameter gradient updates are selected and pulled from the parameter service library to obtain the aggregation result; this transmission strategy requires worker nodes to calculate the similarity of adjacent iteration gradients before each gradient transmission, and to filter out the parts that need to be sent by using a preset gradient filtering algorithm with a gradient similarity threshold. The remote central module processes the gradients from different worker nodes and saves copies to replace the similar gradients that were not received, thereby effectively filtering out similar gradients that do not need to be sent without disrupting the aggregation logic;
[0060] Dynamic regulation:
[0061] S04. Monitor and analyze the loss change trend of the training model, and adjust the gradient similarity threshold based on the loss change trend; specifically, periodically monitor the training loss value of the distributed training model; calculate the loss value difference between every two iterations based on the training loss value; and determine the loss change trend of the distributed training model based on the loss value difference.
[0062] If the difference is less than or equal to 0, it is considered that the model loss value of the distributed training model has not decreased; the gradient similarity threshold is adjusted to the first gradient similarity threshold based on the controller module. In this embodiment, the first gradient similarity threshold is 0.9. The value of 0.9 is a conventional choice for those skilled in the art in training, and it is believed that a threshold of 0.9 will not affect the model training and reduce gradient loss.
[0063] If the difference is greater than 0 for three consecutive times, the model loss of the distributed training model is considered to have decreased. The gradient similarity threshold is dynamically lowered based on the controller module to reduce communication.
[0064] In this embodiment, the dynamic down-adjustment formula is as follows:
[0065] S t =S t-1 -adjust_step
[0066] Where S t and S t-1 These are the gradient similarity thresholds before and after adjustment, respectively, and adjust_step is a pre-set fixed adjustment step size.
[0067] This dynamic adjustment method uses a gradient similarity threshold dynamic adjustment strategy to optimize overall training performance. Specifically, we monitor the changes in the training loss value during training to determine the model's convergence status, and then dynamically adjust the threshold accordingly.
[0068] In the technical solutions of the above embodiments, we dynamically balance the system's performance and risk by detecting the model loss value. To test the performance of Embodiment 1 of this application, we constructed a 100G test platform, which includes three servers with P100 GPUs and one CPU server connected to the same rack-top switch. We ran popular DNN training tasks to evaluate the system performance, including VGG19, ResNet101, and AlexNet.
[0069] Figure 4 This is a performance test chart comparing the embodiments of this application with existing distributed training communication schemes (NCCL, OmniReduce, PS). See attached figure. Figure 4Performance comparisons were conducted between this method and existing distributed training communication schemes NCCL, OmniReduce, and PS (based on VGG19, ResNet101, and AlexNet loads) in a 100Gbps network environment with three worker nodes. Results show that this method achieves the best performance in all three model training tasks, with a maximum speedup of 1.18X relative to Nto1 Worker-PS, 1.08X relative to NCCL, and 1.22X relative to OmniReduce. Among these models, the performance improvement of this method is significantly greater for VGG19 and AlexNet training than for ResNet101. ResNet101 has a computationally intensive load, so the communication speedup is not significant. This method outperforms NCCL and Nto1Worker-PS schemes in all model tasks. Among them, our method outperforms OmniReduce in VGG19, ResNet101 and AlexNet models. This is because OmniReduce is mainly based on the optimization of model sparsity, while the gradient sparsity of these three models is relatively low. Therefore, our method shows better performance in aggregation compared with existing solutions.
[0070] It should be noted that the above formulas are all derived from software simulations using a large amount of data, and a formula that is close to the actual value is selected. The coefficients in the formulas are set by those skilled in the art based on the actual situation.
[0071] Continue to refer to the appendix Figure 4 The comparison shows that, compared with the basic remote central module architecture, this invention improves training throughput by up to 1.18X and training speed by up to 1.15X by transmitting fewer gradients. Compared with the efficient sparse communication scheme OmniReduce, it improves training throughput by up to 1.22X and training speed by up to 1.18X, and can even avoid the performance loss caused by low model sparsity; compared with the most popular Ring-AllReduce scheme NCCL, it can also achieve a throughput improvement of 1.08X and a speedup of 1.07X.
[0072] In summary, the distributed training communication acceleration method based on similar gradient filtering provided in Embodiment 1 of this invention is very effective for the performance of distributed training models. It can reduce network traffic by selecting transmissions and accelerate training while ensuring model quality by relying on a threshold dynamic adjustment algorithm, which greatly improves the performance of distributed training communication and protects the robustness of model training while minimizing communication volume.
[0073] Example 2
[0074] Embodiment 2 of this application provides a distributed training communication acceleration system based on similarity gradient filtering. Figure 2This is a diagram illustrating a distributed training communication acceleration system architecture based on similarity gradient filtering, according to an exemplary embodiment; see attached diagram. Figure 2 The system includes at least three key components and a worker module;
[0075] The worker module includes several worker nodes, which are used to collect the parameter gradients of each worker node in the training model.
[0076] The transmitter module is used to import the parameter gradient into a preset gradient filtering algorithm and filter the gradient subset according to the gradient similarity threshold of the gradient filtering algorithm.
[0077] The remote center module is pre-configured with a parameter service library to record the latest parameter gradient copies of each working node. This allows for querying and retrieving parameter gradients during gradient aggregation. Based on the gradient subset and the parameter service library, the aggregation result is updated and returned to the working node.
[0078] The controller module is used to monitor and analyze the loss change trend of the trained model, and adaptively adjust the gradient similarity threshold based on the loss change trend.
[0079] For each worker node, only incremental gradients are sent during each gradient aggregation. Similar gradients that do not need to be sent are provided directly by the remote central module on the server side. The worker node uses the transmitter module to filter out the gradients that need to be sent and forces the sending of gradients when necessary to compensate for gradient loss. During training, the controller module monitors changes in the loss value and dynamically adjusts the gradient similarity threshold in the transmitter module based on its trend to avoid compromising model accuracy and convergence speed. The components within this system work collaboratively to protect the robustness of model training while minimizing communication overhead.
[0080] Example 3
[0081] Embodiment 3 of this application provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it implements the distributed training communication acceleration method based on similar gradient filtering provided in the first aspect above.
[0082] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.
[0083] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, is used to cause the terminal device to execute the distributed training communication acceleration method based on similar gradient filtering in Embodiment 1.
[0084] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.
[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0086] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A distributed training communication acceleration method based on similarity gradient filtering, characterized in that, The method includes: Collect the parameter gradients of each working node of the training model; The parameter gradient is imported into a preset gradient filtering algorithm, and a gradient subset is obtained by filtering according to the gradient similarity threshold of the gradient filtering algorithm. Based on the gradient subset and the pre-configured parameter service library, the aggregation result is updated and returned to the working node; Monitor and analyze the loss change trend of the trained model, and adjust the gradient similarity threshold based on the loss change trend.
2. The distributed training communication acceleration method based on similarity gradient filtering according to claim 1, characterized in that, The step of importing the parameter gradient into a preset gradient filtering algorithm and obtaining a gradient subset based on the gradient similarity threshold of the gradient filtering algorithm includes: The working node calculates the similarity of adjacent iteration gradients based on similarity evaluation to obtain the first parameter gradient; The gradient of the first parameter is imported into a preset gradient filtering algorithm to obtain the necessary gradient of the second parameter, and the gradient of the second parameter is integrated into a gradient subset.
3. The distributed training communication acceleration method based on similarity gradient filtering according to claim 2, characterized in that, If the second parameter gradient is not found during the filtering process, a first notification is sent to the parameter service library.
4. The distributed training communication acceleration method based on similarity gradient filtering according to claim 1, characterized in that, The process of updating the aggregated result based on the gradient subset and the pre-configured parameter service library includes: A pre-configured parameter service library receives the gradient subsets from different working nodes; Based on the header field within the gradient subset, parameter gradient updates are retrieved from the parameter service library to obtain the aggregated result.
5. The distributed training communication acceleration method based on similarity gradient filtering according to claim 1, characterized in that, The monitoring and analysis of the loss change trend of the trained model includes: Regularly monitor the training loss value of the distributed training model; Calculate the difference in loss value between every two iterations based on the training loss value; The trend of loss change in the distributed training model is determined based on the difference in the loss values. If the difference is less than or equal to 0, it is considered that the model loss value of the distributed training model has not decreased; If the difference is greater than 0 for three consecutive times, the model loss value of the distributed training model is considered to have decreased.
6. The distributed training communication acceleration method based on similarity gradient filtering according to claim 5, characterized in that, Adjusting the gradient similarity threshold based on the loss change trend includes: If the difference is less than or equal to 0, the gradient similarity threshold is adjusted to the first gradient similarity threshold to reduce gradient loss. If the difference is greater than 0 for three consecutive times, the gradient similarity threshold is dynamically lowered to reduce communication volume. The dynamic lowering formula is as follows: S t =S t-1 -adjust_step Where S t and S t-1 These are the gradient similarity thresholds before and after adjustment, respectively, and adjust_step is a pre-set fixed adjustment step size.
7. The distributed training communication acceleration method based on similarity gradient filtering according to claim 6, characterized in that, The first gradient similarity threshold is 0.
9.
8. A distributed training communication acceleration system based on similarity gradient filtering, characterized in that, The system includes: The worker module includes several worker nodes, which are used to collect the parameter gradients of each worker node in the training model; The transmitter module is used to import the parameter gradient into a preset gradient filtering algorithm and filter the gradient subset according to the gradient similarity threshold of the gradient filtering algorithm. The remote center module is pre-configured with a parameter service library to record the latest parameter gradient copies of each working node. This allows for querying and retrieving parameter gradients during gradient aggregation. Based on the gradient subset and the parameter service library, the aggregation result is updated and returned to the working node. The controller module is used to monitor and analyze the loss change trend of the trained model, and adjust the gradient similarity threshold based on the loss change trend.
9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the distributed training communication acceleration method based on similar gradient filtering as described in any one of claims 1-8.