Efficient data parallel training method under high-delay low-bandwidth scene

Abstract

The invention discloses an efficient data parallel training method in a high-delay low-bandwidth scene, and the method comprises the steps: constructing a communication-calculation tradeoff model based on a real-time network state through dynamic combined optimization of a gradient compression ratio and a delay step length, and carrying out the self-adaptive adjustment of parameters, so as to balance the precision and efficiency; an error feedback mechanism is used for compensating compression loss, and precision attenuation under the high compression rate is reduced; the synchronization frequency is controlled in combination with a delay gradient aggregation strategy, and calculation and communication parallelization is achieved; network conditions are periodically monitored, parameters are updated, and the cooperative defect of a traditional independent optimization strategy in a dynamic environment is overcome. Experiments show that under the conditions of 32 nodes, 100 Mbps bandwidth and 500 ms delay, compared with a D-SGD method and a DD-EF-SGD method, the training efficiency of the method is improved by 5.07 times and 1.24 times respectively, and the distributed training performance of communication limited scenes such as a cross-wide area network is remarkably optimized.

Description

Technical Field

[0001] The present invention relates to distributed machine learning technology, and in particular to an efficient data parallel training method in high-latency and low-bandwidth scenarios. Background Art

[0002] With the rapid evolution of deep learning technology, the number of model parameters and the volume of training data continue to grow. Single-node computing resources are no longer sufficient for large-scale model training. Distributed machine learning has emerged as a response to this challenge. By distributing computing tasks in parallel across multiple compute nodes, it effectively accelerates training and improves resource utilization. Currently, Distributed Stochastic Gradient Descent (D-SGD) is the most commonly used distributed optimization algorithm.

[0003] However, D-SGD faces significant challenges across wide-area networks, such as communication bottlenecks and high end-to-end latency. For example, latency between the US and Australia can be as high as 200 milliseconds, and bandwidth can be less than 10 Gbps. When training models using distributed computing resources from the US to Australia, communication issues can reduce training efficiency by at least 40%.

[0004] To alleviate the impact of communication bottlenecks, existing research has proposed the following two main optimization strategies: 1) Gradient compression: By transmitting only a portion of the gradient vector (e.g., Top-k sparsification), communication overhead can be significantly reduced. This strategy is often combined with an error feedback mechanism to record and accumulate information lost during the compression process, compensating for it in subsequent iterations to maintain training accuracy. However, overly aggressive compression (e.g., a compression ratio of less than 1%) can significantly degrade model accuracy, making it difficult to maintain stable control, especially when network conditions fluctuate drastically.

[0005] 2) Delayed aggregation: This allows for synchronous gradient updates after multiple iterations, effectively masking communication delays and improving computing resource utilization. This strategy allows for partial overlap between computation and communication, improving training throughput within latency tolerance. However, delayed gradient usage can introduce staleness, impacting training convergence speed and model accuracy.

[0006] While the two aforementioned strategies have achieved some optimization results in their respective directions, existing methods mostly apply and adjust them independently, overlooking the complex interactions between them when used together. Recent research has demonstrated a significant interaction between gradient compression and delayed aggregation: delays amplify the negative impact of compression errors, while inappropriate compression ratios can weaken the benefits of delayed strategies. Consequently, in dynamically changing communication environments, traditional independent optimization strategies often fail to achieve optimal training efficiency and model performance.

[0007] The existing technologies and their shortcomings are summarized as follows: 1) Distributed Stochastic Gradient Descent (D-SGD) D-SGD is the most basic distributed training strategy. In each round of training, each worker node computes gradients using local data, and then all nodes synchronously communicate and aggregate these gradients to update the global model parameters.

[0008] Disadvantages: Frequent communication and high bandwidth overhead; no compression and delay mechanism, extremely low training efficiency in communication-restricted scenarios; high requirements for inter-node synchronization, easily affected by slow nodes.

[0009] 2) Adaptive Gradient Compression Strategy (Accordion) Accordion is a gradient compression method based on identifying key learning stages. It dynamically adjusts the compression ratio, using lower compression (for higher accuracy) in the early stages of training or during key stages, and higher compression (for lower communication overhead) in the remaining stages to reduce bandwidth pressure. Disadvantages: Compression ratio adjustment relies on heuristic rules and cannot adapt to complex network fluctuations; it does not support co-optimization with delay strategies; and its adaptability to different models and tasks is limited.

[0010] 3) DGA (DelayedGradientAveraging) DGA is an optimization method that introduces delayed aggregation. Each node can upload gradients after multiple rounds of local training. The server asynchronously aggregates gradient values ​​at different times, effectively masking communication delays and improving throughput. Disadvantages: Delay introduces gradient staleness, potentially resulting in reduced accuracy; it is not designed in conjunction with gradient compression strategies; and the delay parameter must be manually set, lacking adaptability.

[0011] 4) DD-EF-SGD (Delay+Error-Feedback+Compression) DD-EF-SGD combines delayed aggregation with gradient compression (with error feedback) to improve communication efficiency while maintaining compression accuracy. Each node uploads historical gradients after compression and feedback, which are then aggregated by the central server after a certain number of delay steps. Disadvantages: It uses a fixed compression ratio δ and delay step size τ; it cannot be dynamically adjusted based on the actual network; it ignores the coupling relationship between delay and compression error; and its combined effectiveness is limited, potentially leading to unstable training.

[0012] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0013] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide an efficient data parallel training method in a high-latency and low-bandwidth scenario.

[0014] To achieve the above object, the present invention adopts the following technical solutions: An efficient data parallel training method in a high-latency, low-bandwidth scenario includes the following steps: S1. Dynamic Parameter Joint Optimization: Based on the current network bandwidth and latency parameters, a coupled relationship model between gradient compression rate and latency step size is constructed. By minimizing the communication-computation tradeoff objective function, the optimal gradient compression rate and latency step size are calculated in real time. S2. Gradient Compression and Error Feedback: Compresses locally calculated gradients, records the compression error, and accumulates it for compensation in subsequent iterations to maintain model training accuracy. S3. Delayed Gradient Aggregation: Based on the optimized delay step size, the frequency of gradient synchronization updates is controlled, allowing computation and communication to be performed in parallel to reduce latency. S4. Adaptive cyclic update: Periodically monitor network state parameters, re-execute dynamic parameter joint optimization, and update the gradient compression rate and delay step size until the model converges.

[0015] Furthermore, in step S1, the dynamic parameter joint optimization specifically includes: It traverses the range of delay steps that satisfy the communication and computation time constraints, calculates candidate gradient compression rates based on the time budget and model size, evaluates each parameter combination through the objective function, and selects the optimal solution that minimizes the end-to-end training time.

[0016] Furthermore, the objective function is defined as a nonlinear function of the gradient compression rate and the delay step size, and its optimization goal is to minimize the total communication and computation time while ensuring the accuracy of the model.

[0017] Furthermore, the dynamic parameter joint optimization further includes: The preset delay step range is traversed, and the corresponding gradient compression ratio is calculated for each candidate delay step to ensure that it meets the communication and computation time constraints. The communication-computation efficiency of each parameter combination is evaluated through a nonlinear objective function, and the delay step and compression ratio that minimize the objective function value are iteratively selected as the global optimization parameters.

[0018] Furthermore, in step S2, the gradient compression is achieved by sparsification, quantization or low-rank approximation, the compressed gradient is superimposed with the historical error buffer and then transmitted, and the local error buffer is updated after global aggregation.

[0019] Furthermore, in step S3, the delayed gradient aggregation improves resource utilization by allowing gradients to be synchronized after multiple local iterations, combining overlapping execution of computation and communication; the upper limit of the delay step is determined by the constraint relationship between communication time and computation time.

[0020] Furthermore, in step S4, the adaptive cyclic update specifically includes: regularly collecting network bandwidth and delay parameters, triggering the dynamic parameter optimization module, and recalculating and updating the global gradient compression rate and delay step size to adapt to fluctuations in network conditions.

[0021] Furthermore, the method further comprises: In the global model update phase, weighted aggregation is performed based on the compressed gradients and node weights to generate updated global model parameters, which are then distributed synchronously to all computing nodes. The weighted aggregation is performed by multiplying the compressed gradient increments of each node by the preset node weights and then accumulating them, and the global model parameters are adjusted in combination with the learning rate.

[0022] Furthermore, the global model updating stage further includes: By utilizing the error feedback, the gradient information lost during the compression process is accumulated into the local gradient calculation of the next iteration to offset the accuracy loss introduced by the compression.

[0023] Furthermore, the error feedback includes: superimposing the current gradient with the historical error buffer and then compressing and transmitting the superimposed gradient, and updating the local error buffer according to the compressed gradient increment to compensate for the compression loss.

[0024] The present invention has the following beneficial effects: The present invention provides an efficient data parallel training method in high-latency and low-bandwidth scenarios. By systematically jointly optimizing gradient compression and delay aggregation strategies, a mathematical model based on the dynamic changes of network bandwidth and delay is constructed. The gradient compression rate (δ) and delay step size (τ) are calculated and adjusted in real time, effectively balancing model accuracy and training efficiency while reducing communication overhead. By introducing an error feedback mechanism to continuously compensate for the gradient information lost during the compression process, the problem of accuracy degradation caused by high compression rates in traditional gradient compression methods is significantly alleviated. Combined with a dynamic adaptive algorithm to periodically monitor network status and update parameters, the defect of existing independent optimization strategies that lack collaborative adjustment capabilities under complex network fluctuations is solved, thereby achieving efficient parallelization of computing and communication in a high-latency, low-bandwidth cross-wide area network environment. Experimental data show that in a typical scenario with 32 nodes, 100Mbps bandwidth and 500ms latency, the present invention achieves training accelerations of up to 5.07 times and 1.24 times compared to traditional D-SGD and DD-EF-SGD methods, respectively, while maintaining model accuracy. This fully demonstrates its robustness and adaptability under dynamic network conditions, as well as its significant optimization effect in scenarios with limited communication resources, providing an efficient and stable solution for large-scale distributed deep learning training.

[0025] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an overall flow chart of the efficient data parallel training method in the high-latency and low-bandwidth scenario of the present invention.

[0027] Figure 2 This is an overall framework diagram of the algorithm of an embodiment of the present invention.

[0028] Figure 3 Schematic diagram of delayed polymerization according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0031] In existing distributed deep learning training frameworks, as models scale and the number of nodes increase, insufficient communication bandwidth and end-to-end latency become core bottlenecks that severely restrict training efficiency. Particularly in heterogeneous or cross-regional network environments with limited communication, traditional distributed stochastic gradient descent methods face the following technical challenges when transmitting gradient information: 1. Gradient compression introduces accuracy loss: Existing gradient sparsification compression methods often lose gradient information while reducing the amount of communication data. Especially at high compression ratios (e.g., <1%), model convergence speed decreases significantly or even fails to converge. 2. Delayed aggregation leads to gradient staleness: While delayed aggregation strategies can improve GPU utilization and alleviate congestion caused by communication latency, they can also reduce model accuracy by using "outdated" gradients, impacting final training results. 3. Lack of coordinated optimization mechanisms for compression and delay strategies: Most current mainstream training optimization methods design and adjust parameters for gradient compression and delayed aggregation separately, failing to consider their coupled effects during training. Independent optimization often leads to an imbalanced training process and fails to achieve an optimal trade-off between accuracy, speed, and communication costs. 4. Lack of adaptability under dynamic network conditions: In dynamic network scenarios such as cross-region, multi-cloud, and multi-node, bandwidth and latency can change at any time. Existing training frameworks generally use static compression ratios or fixed delay parameters, which cannot be adjusted in real time according to network conditions, resulting in a significant decrease in training efficiency.

[0032] To address these problems, the present invention proposes a coordinated optimization mechanism by establishing a mathematical model and jointly analyzing the parameters of gradient compression and delayed aggregation. The mechanism aims to adjust training parameters in real time to optimize communication efficiency and training performance, thereby significantly improving training efficiency and model accuracy, and ultimately achieving a more efficient distributed training framework.

[0033] In order to achieve the optimal trade-off, it is necessary to establish the relationship between end-to-end time and model accuracy, while taking into account gradient compression and delay aggregation. To meet this challenge, the present invention proposes an efficient distributed training framework based on the joint optimization of gradient compression and delay aggregation, which is suitable for communication-constrained environments. By uniformly modeling the delay staleness and compression error in the training convergence process, a mathematical coupling relationship between the gradient compression rate and the delay parameter is constructed. On this basis, the framework introduces an adaptive joint optimization algorithm, which can calculate the optimal compression ratio and delay step size in real time according to the dynamic changes in network bandwidth and delay, thereby achieving the optimal balance between model accuracy and training speed. This invention is not only robust under dynamic network conditions, but also significantly reduces the communication load, effectively improving the overall performance of distributed training in resource-constrained scenarios.

[0034] See Figure 1 , an embodiment of the present invention provides an efficient data parallel training method in a high-latency and low-bandwidth scenario, comprising the following steps: Step S1. Dynamic parameter joint optimization: Based on the current network bandwidth and delay parameters, a coupling relationship model between the gradient compression rate (δ) and the delay step size (τ) is constructed. By minimizing the communication-computation trade-off objective function, the optimal gradient compression rate and delay step size are calculated in real time.

[0035] In some embodiments, in step S1, the dynamic parameter joint optimization specifically includes: traversing the delay step range that meets the communication and computing time constraints, calculating the candidate gradient compression rate according to the time budget and model scale, evaluating each parameter combination through the objective function, and selecting the optimal solution that minimizes the end-to-end training time.

[0036] In some embodiments, the objective function is defined as a nonlinear function of gradient compression rate and delay step size, and its optimization goal is to minimize the total communication and calculation time while ensuring model accuracy.

[0037] In some embodiments, the dynamic parameter joint optimization further includes: traversing a preset delay step range, calculating the corresponding gradient compression rate for each candidate delay step to ensure that it meets the communication and computing time constraints; evaluating the communication-computation efficiency of each parameter combination through a nonlinear objective function, and iteratively selecting the delay step and compression rate that minimizes the objective function value as the global optimization parameters.

[0038] Step S2. Gradient compression and error feedback: Compress the locally calculated gradients, record the compression errors, and accumulate them for compensation in subsequent iterations to maintain model training accuracy.

[0039] In some embodiments, in step S2, the gradient compression is achieved by sparsification, quantization or low-rank approximation, the compressed gradient is superimposed with the historical error buffer and then transmitted, and the local error buffer is updated after global aggregation.

[0040] Step S3. Delayed gradient aggregation: Based on the optimized delay step size, the frequency of gradient synchronization updates is controlled so that computation and communication are performed in parallel to reduce waiting time.

[0041] In some embodiments, in step S3, the delayed gradient aggregation improves resource utilization by allowing gradients to be synchronized after multiple local iterations, combining overlapping execution of computation and communication; the upper limit of the delay step is determined by the constraint relationship between communication time and computation time.

[0042] Step S4. Adaptive loop update: Periodically monitor network state parameters, re-execute dynamic parameter joint optimization, and update the gradient compression rate and delay step size until the model converges.

[0043] In some embodiments, in step S4, the adaptive cyclic update specifically includes: regularly collecting network bandwidth and delay parameters, triggering a dynamic parameter optimization module, and recalculating and updating a global gradient compression rate and a delay step size to adapt to fluctuations in network conditions.

[0044] In some embodiments, the method further includes: in the global model update stage, weighted aggregation is performed based on the compressed gradient and node weight to generate updated global model parameters, and synchronously distribute them to all computing nodes; the weighted aggregation is performed by multiplying the compressed gradient increment of each node with the preset node weight and accumulating them, and adjusting the global model parameters in combination with the learning rate.

[0045] In some embodiments, the global model update stage further includes: utilizing the error feedback to accumulate the gradient information lost during the compression process into the local gradient calculation of the next round of iteration to offset the accuracy loss introduced by the compression.

[0046] Furthermore, the error feedback includes: superimposing the current gradient with the historical error buffer and then compressing and transmitting the superimposed gradient, and updating the local error buffer according to the compressed gradient increment to compensate for the compression loss.

[0047] The following further describes specific embodiments of the present invention, its algorithm examples and experimental verification.

[0048] This paper addresses the communication bottleneck problem in existing distributed training by comprehensively considering the coupling relationship between gradient compression and delay aggregation, and proposes a joint optimization strategy. During the training process, this strategy can dynamically adjust the gradient compression rate (δ) and delay step size (τ) according to the current network bandwidth and delay conditions, thereby reducing communication overhead while ensuring model accuracy and training convergence speed, significantly improving the efficiency of distributed training in communication-constrained scenarios. The key goal of this algorithm is to minimize an objective function that measures the communication-computation trade-off. , in order to achieve minimal end-to-end training time while ensuring no loss of model accuracy.

[0049] Figure 2 The overall algorithm framework of the embodiment of the present invention is presented. The roles of computing nodes can be divided into two categories: worker nodes and server nodes. These two types of nodes work together to train machine learning models. The specific implementation steps of the embodiment of the present invention include: Each node performs the following process in each training iteration: 1) Calculate gradients using local data; 2) The sum of the gradient and the error buffer is input into the compressor for sparse compression; 3) Update the error feedback cache; 4) Based on the set τ value, decide whether to upload the current compressed gradient to the global state; The scheduling module runs the joint optimization algorithm 1 to calculate the optimal (δ, τ) based on the current network status; The global model is updated based on the aggregated gradient and synchronized to each node; Repeat the above process until the model converges or the termination condition is met.

[0050] Gradient compression algorithms reduce the communication overhead of gradient transmission by compressing the gradient (such as quantization, sparsification, or low-rank approximation), converting the original high-dimensional gradient data into a more compact representation, thereby significantly reducing the amount of data transmitted.

[0051] The delayed aggregation algorithm allows the gradients of some computing nodes to participate in the aggregation of the global gradient after a certain delay, thereby making communication and calculation parallel. Figure 3 If the delay step is too long, the gradient update value will be outdated, which will affect the model accuracy. If the delay step is too short, when the bandwidth is too low or the transmission volume is too large, the gradient cannot be transmitted completely within the delay time, resulting in a waiting phenomenon.

[0052] The algorithm accepts four inputs: global model size S g , local calculation time per step , global synchronization communication overhead b , and calculation time First, we iterate over the possible delay steps t , which ranges from the minimum required to meet communication and computation time constraints = to the maximum value = For each candidate t , the algorithm calculates the corresponding compression ratio based on the time budget and model size d , ensuring that it does not exceed the maximum value allowed by the system computing power and does not exceed 1. Then according to d and t Calculate the optimization objective function: .

[0053] Finally found Minimum parameter combination The specific algorithm example is shown in Table 1 in the pseudo code form of Algorithm 1.

[0054] Table 1

[0055] The above algorithm is applied to distributed training. The specific process is shown in Table 2 in the pseudo code form of Algorithm 2. During the training process, the network bandwidth parameters are monitored periodically (every E rounds) and transmission delay parameters b , and dynamically update using Algorithm 1 t and d Then, based on the new t and d , compress the gradient of each node and delay its transmission, then calculate and update the local error and global model parameters.

[0056] Table 2

[0057] Examples: Example 1: Large-scale deep learning model training across a wide area network environment.

[0058] In a cross-wide area network environment, distributed training usually faces high communication overhead, especially when training large-scale models. Traditional distributed training methods may lead to inefficient utilization of computing resources, and in a high-latency network environment, frequent gradient synchronization may cause significant waiting time, thereby affecting training efficiency. In order to overcome these problems, the present invention combines gradient compression with delayed aggregation strategies to effectively reduce communication overhead and speed up the calculation process. Gradient compression alleviates the bandwidth bottleneck by reducing the amount of gradient data transmitted; while delayed aggregation reduces the waiting time between nodes by dividing the computing task into multiple stages and performing calculations and data transmission in parallel. By adjusting the compression ratio and parallel strategy in real time, the present invention can significantly improve training efficiency while ensuring model accuracy, and is particularly suitable for large-scale deep learning training tasks in a cross-wide area network environment.

[0059] Example 2: Distributed training applications in intelligent manufacturing.

[0060] In the field of intelligent manufacturing, industrial equipment and sensors generate a large amount of data, which needs to be analyzed and trained in real time through distributed systems. However, traditional training frameworks are often unable to effectively cope with high bandwidth requirements when processing these large-scale data flows, resulting in system bottlenecks and inefficient data transmission. The present invention significantly reduces the bandwidth pressure of large-scale data transmission by applying gradient compression technology in a cross-wide area network environment, and at the same time accelerates the calculation process by combining a delay aggregation strategy. This makes data exchange and computing processing between devices more efficient and reduces delays caused by synchronous waiting. In addition, the present invention also has an adaptive adjustment mechanism that can dynamically optimize training parameters according to real-time bandwidth and delay changes across the wide area network, ensuring that the system processing speed is improved while maintaining model accuracy, thereby greatly improving the efficiency of distributed training in the field of intelligent manufacturing.

[0061] In order to solve the problems existing in the prior art, the present invention systematically conducts mathematical modeling and joint optimization of gradient compression and delayed aggregation for the first time, and has a theoretically supported optimal parameter search mechanism ( Optimization) and dynamic self-adaptation significantly improve training efficiency and accuracy in communication-constrained environments, representing a significant improvement and supplement to existing methods. Compared to other methods, this invention can more flexibly adapt to bandwidth limitations and latency fluctuations across wide-area networks, providing a more efficient and stable distributed training solution. It is particularly suitable for cross-wide-area network training, cloud computing environments, and other scenarios with unstable network conditions.

[0062] A comparison of the training efficiency of the present invention and other comparative technical solutions is shown in Table 3. The numbers are the training time (in seconds) required to achieve a given perplexity / accuracy, where the multiplier in the brackets represents the multiple of the algorithm's time compared to DeCo-SGD, i.e., the speed improvement.

[0063] Table 3: Differences and similarities between the present invention and other comparative technical solutions

[0064] This invention significantly accelerates the distributed training process without sacrificing accuracy. As shown in the results in Tables 4 and 5, the network bandwidth was set to 100 Mbps and the end-to-end communication latency was set to 500 ms. The performance of the invention was tested with different numbers of nodes n (up to 32). The table below shows the training time (in seconds) required for different algorithms to achieve target performance on the GPT@Wikitext (using perplexity as the metric) and ViT@ImageNet (using accuracy as the metric) tasks. In the scenario with n = 32, the invention achieved speedups of up to 5.07 times and 1.24 times compared to traditional D-SGD and DD-EF-SGD, respectively.

[0065] Table 4: Comparison of training efficiency of the GPT model using the Wikitext dataset using the proposed method and other methods

[0066] Table 5: Comparison of the training efficiency of the ViT model with other methods on the Imagenet dataset

[0067] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0068] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0069] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0070] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disc or a read-only optical disc (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0071] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0072] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0074] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0075] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0076] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0077] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0078] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0079] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. An efficient data parallel training method in high-latency and low-bandwidth scenarios, characterized by: The following steps are involved: S1. Dynamic Parameter Joint Optimization: Based on the current network bandwidth and latency parameters, a coupled relationship model between gradient compression rate and latency step size is constructed. By minimizing the communication-computation tradeoff objective function, the optimal gradient compression rate and latency step size are calculated in real time. S2. Gradient Compression and Error Feedback: Compresses locally calculated gradients, records the compression error, and accumulates it for compensation in subsequent iterations to maintain model training accuracy. S3. Delayed Gradient Aggregation: Based on the optimized delay step size, the frequency of gradient synchronization updates is controlled, allowing computation and communication to be performed in parallel to reduce latency. S4. Adaptive cyclic update: Periodically monitor network state parameters, re-execute dynamic parameter joint optimization, and update the gradient compression rate and delay step size until the model converges.

2. The efficient data parallel training method according to claim 1, characterized in that: In step S1, the dynamic parameter joint optimization specifically includes: It traverses the range of delay steps that satisfy the communication and computation time constraints, calculates candidate gradient compression rates based on the time budget and model size, evaluates each parameter combination through the objective function, and selects the optimal solution that minimizes the end-to-end training time.

3. The efficient data parallel training method according to claim 2, characterized in that: The objective function is defined as a nonlinear function of the gradient compression rate and the delay step size, and its optimization goal is to minimize the total communication and calculation time while ensuring the accuracy of the model.

4. The efficient data parallel training method according to claim 3, wherein: The dynamic parameter joint optimization further includes: The preset delay step range is traversed, and the corresponding gradient compression ratio is calculated for each candidate delay step to ensure that it meets the communication and computation time constraints. The communication-computation efficiency of each parameter combination is evaluated through a nonlinear objective function, and the delay step and compression ratio that minimize the objective function value are iteratively selected as the global optimization parameters.

5. The efficient data parallel training method according to claim 1, wherein: In step S2, the gradient compression is achieved by sparsification, quantization or low-rank approximation, the compressed gradient is superimposed with the historical error buffer and then transmitted, and the local error buffer is updated after global aggregation.

6. The efficient data parallel training method according to claim 1, wherein: In step S3, the delayed gradient aggregation improves resource utilization by allowing gradients to be synchronized after multiple local iterations and combining the overlapping execution of computation and communication. The upper limit of the delay step is determined by the constraint relationship between communication time and computation time.

7. The efficient data parallel training method according to claim 1, wherein: In step S4, the adaptive cyclic update specifically includes: regularly collecting network bandwidth and delay parameters, triggering the dynamic parameter optimization module, and recalculating and updating the global gradient compression rate and delay step size to adapt to fluctuations in network conditions.

8. The efficient data parallel training method according to claim 1, wherein: The method further comprises: In the global model update phase, weighted aggregation is performed based on the compressed gradients and node weights to generate updated global model parameters, which are then distributed synchronously to all computing nodes. The weighted aggregation is performed by multiplying the compressed gradient increments of each node by the preset node weights and then accumulating them, and the global model parameters are adjusted in combination with the learning rate.

9. The efficient data parallel training method according to claim 8, wherein: The global model updating stage further includes: By utilizing the error feedback, the gradient information lost during the compression process is accumulated into the local gradient calculation of the next iteration to offset the accuracy loss introduced by the compression.

10. The efficient data parallel training method according to claim 1, wherein: The error feedback includes: superimposing the current gradient with the historical error buffer and then compressing and transmitting it, and updating the local error buffer according to the compressed gradient increment to compensate for the compression loss.

Images