A gradient synchronization method in distributed training oriented to a layered heterogeneous network topology
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- EAST CHINA NORMAL UNIV
- Filing Date
- 2026-03-06
- Publication Date
- 2026-06-05
Smart Images
Figure CN122160380A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of distributed computing, deep learning model training, and network communication scheduling, and particularly to a gradient synchronization method for distributed training in hierarchical heterogeneous network topologies. Background Technology
[0002] As the scale and training data volume of deep learning neural networks continue to grow, single-machine, single-GPU environments can no longer complete model training within an acceptable timeframe. To shorten training time, the industry commonly adopts distributed parallel training, with data parallelism being the most widely used parallel training mode. In data parallelism, multiple copies of the same model are deployed on different computing nodes, each independently processing a different subset of training data. After completing forward and backward propagation, gradient synchronization is needed to maintain parameter consistency, and gradient synchronization overhead has become one of the key factors affecting the efficiency of distributed training.
[0003] Existing gradient synchronization methods mainly fall into two categories: parameter server architecture and all-reduce architecture. In the parameter server architecture, worker nodes send their gradients to the parameter server after completing local gradient calculations. The parameter server then performs gradient aggregation and parameter updates, and distributes the updated results back to the worker nodes. This architecture is simple to implement, but the parameter server nodes are prone to becoming a concentrated bottleneck in communication and computation. As the number of worker nodes and the model size increase, its network bandwidth and computing power struggle to support linear system scaling. The all-reduce architecture achieves gradient synchronization by building a peer-to-peer communication topology among all participating computation nodes. Typical algorithms include ring all-reduce and tree-based all-reduce, which can distribute the communication load to some extent, but typically assume that the underlying network is isomorphic and the topology is relatively simple.
[0004] Real-world data center networks often exhibit distinct hierarchical and heterogeneous characteristics: within a single server, compute nodes are connected via peripheral interconnect buses or dedicated high-speed interconnects; between servers, they are interconnected through rack switches, core switches, and potentially cross-data center links, among other multi-level network devices. Significant differences exist between these different levels in terms of bandwidth and latency. While existing technologies have developed simple hierarchical full reduction methods, such as performing an intra-machine full reduction first, followed by a smaller-scale full reduction between servers, these methods typically only roughly distinguish between "intra-machine" and "inter-machine" levels. They fail to accurately characterize the multi-level network structure within and between racks, leading to underutilization of some links and the potential for other links to become bottlenecks.
[0005] Regarding the scheduling of gradient synchronization tasks, existing methods often optimize based on a single gradient reduction operation, frequently making local improvements to a specific communication algorithm or library, lacking a unified scheduling perspective across model layers, gradient tensors, and network layers. In actual training, different model layers have varying degrees of impact on iteration time. Some earlier model layers, if they can complete gradient synchronization and update parameters early, can start the forward computation of the next iteration earlier, while some later model layers, even with slight delays in gradient synchronization, have a relatively small impact on the overall iteration time. However, without a unified scheduling mechanism, gradient synchronization is usually executed in a fixed order, making it impossible to flexibly adjust priorities based on model structure and network state, and hindering the full exploitation of potential parallelism.
[0006] In summary, existing data-parallel gradient synchronization methods still suffer from problems such as insufficient topology awareness, inadequate utilization of layers, lack of unified scheduling across layers, and insufficient overlap between computation and communication in hierarchical heterogeneous network topologies. It is necessary to propose a new gradient synchronization scheduling method that can fully utilize the characteristics of network topology while coordinating the execution order and parallel relationship of different gradient synchronization tasks from a global perspective, so as to improve the overall efficiency of large-scale distributed training. Summary of the Invention
[0007] The purpose of this invention is to address the problems of insufficient topology awareness, inadequate utilization of layers, lack of unified scheduling across layers, and insufficient overlap of computation and communication in existing data-parallel gradient synchronization methods under hierarchical heterogeneous network topologies. This invention proposes a data-parallel gradient synchronization scheduling method for hierarchical heterogeneous network topologies. While ensuring the correctness of training results, it fully utilizes the bandwidth and parallel capabilities of different network layers, coordinating the execution order and parallel relationships of various gradient synchronization tasks from a global perspective. This reduces gradient synchronization overhead and improves the overall efficiency and scalability of large-scale distributed training.
[0008] To achieve the above objectives, this invention provides a gradient synchronization method for distributed training in hierarchical heterogeneous network topologies, comprising the following steps:
[0009] Obtain the network topology information of the distributed training cluster, and construct a hierarchical communication topology model containing multiple network layers based on the physical connection relationship between training nodes;
[0010] The gradient tensors of each layer of the neural network model to be trained are split into multiple gradient blocks according to a preset rule;
[0011] Based on the hierarchical communication topology model, the gradient block synchronization operation is divided into multiple gradient synchronization subtasks;
[0012] According to a preset scheduling strategy, the gradient synchronization subtask is scheduled to a communication library pre-configured on the training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster.
[0013] The gradient blocks in other training nodes of the distributed training cluster are obtained through the communication library.
[0014] Furthermore, the construction of a hierarchical communication topology model containing multiple network layers based on the physical connection relationships between training nodes includes:
[0015] Identify at least one intra-machine interconnection layer and at least one inter-machine network layer based on physical connectivity relationships;
[0016] In the inter-machine network hierarchy, the rack-level, inter-rack level, or higher level are further identified based on the rack and switch connection method of the training node.
[0017] Furthermore, the gradient tensors of each layer of the neural network model to be trained are split into multiple gradient blocks according to a preset rule, including:
[0018] The gradient tensor is divided into several gradient blocks according to the continuous storage address range of the gradient tensor in the memory, and the amount of data corresponding to each gradient block is the same within a preset range.
[0019] Furthermore, based on the hierarchical communication topology model, the gradient block synchronization operation is divided into multiple gradient synchronization subtasks, including:
[0020] A gradient synchronization operation is logically decomposed into a reduction propagation phase and an aggregation collection phase.
[0021] In the hierarchical communication topology model, corresponding reduction dissemination subtasks and aggregation collection subtasks are generated at each network layer, so that each gradient synchronization subtask corresponds to at least a gradient block, a network layer, and a gradient synchronization stage.
[0022] Furthermore, the step of scheduling the gradient synchronization subtask to the communication library pre-configured on the training node according to a preset scheduling strategy includes:
[0023] Based on the hierarchical communication topology model, inter-task dependencies are established for the gradient synchronization subtasks; a preset priority rule is set for the gradient synchronization subtasks, and all gradient synchronization subtasks that have not yet been executed and whose preceding dependency conditions have been met are organized into a global priority queue to be scheduled according to the priority rule.
[0024] Set the total communication resource quota, and set the hierarchical communication resource quota for each network layer of the hierarchical communication topology model;
[0025] Based on the global priority queue to be scheduled, gradient synchronization subtasks that meet the constraints of the total communication resource quota and the corresponding level communication resource quota are selected cyclically from the global priority queue to be scheduled on each training node participating in the training. These subtasks are then submitted to the communication library pre-configured in the training node, and the communication resource quota is dynamically deducted and returned during task execution.
[0026] Furthermore, establishing inter-task dependencies for the gradient synchronization subtasks based on the hierarchical communication topology model includes:
[0027] For the same gradient block, the completion of a reduction dissemination subtask on a certain level network is a prerequisite for the reduction dissemination subtask on a higher level network.
[0028] For the same gradient block, the completion of the aggregation collection subtask on a certain level network is a prerequisite for the aggregation collection subtask on the next lower level network.
[0029] For the same gradient block, the completion of the reduction dissemination subtask at the highest network level is a prerequisite for the aggregation collection subtask at the highest network level.
[0030] The reduction dissemination subtask at the lowest network level has no preconditions.
[0031] Furthermore, a preset priority rule is set for the gradient synchronization subtask, and the priority rule determines the scheduling priority of the gradient synchronization subtask based on at least the following information:
[0032] The neural network model level to which the gradient synchronization subtask belongs is used to give gradient synchronization subtasks in earlier model layers a higher priority than those in later model layers.
[0033] The communication phase to which the gradient synchronization subtask belongs is used to give priority to gradient synchronization subtasks in the reduction and dissemination phase over gradient synchronization subtasks in the aggregation and collection phase.
[0034] The communication topology network hierarchy of gradient synchronization subtasks is used to give priority to gradient synchronization subtasks located at higher network levels than those located at lower network levels.
[0035] Furthermore, after obtaining gradient blocks from other training nodes in the distributed training cluster through the communication library, the process further includes:
[0036] After all the gradient blocks belonging to the same gradient tensor have been acquired, they are reassembled into a complete gradient tensor.
[0037] Update the parameters of the corresponding tensor in the neural network model to be trained in the current training node based on the gradient tensor.
[0038] Through the above mechanism, gradient synchronization subtasks from different model layers and different gradient blocks can be executed interleaved across multiple network layers. Under the premise of ensuring the consistency of communication call order within the same communication group and the correctness of global gradient synchronization results, the time overlap between multi-level communication and forward and backward propagation is achieved, reducing the impact of gradient synchronization on training iteration time.
[0039] Compared with existing technologies, this invention achieves parallel communication and pipeline overlap between different network layers by constructing a hierarchical heterogeneous network topology model, explicitly generating gradient synchronization subtasks across multiple network layers during backpropagation, introducing a unified priority scheduling mechanism based on model hierarchy and network hierarchy, and multi-level communication resource quota control. This enables the gradient synchronization process to make fuller use of intra-machine, inter-machine, and higher-level network resources, effectively reducing the impact of gradient synchronization on training iteration time. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of a hierarchical heterogeneous network topology;
[0041] Figure 2 A schematic diagram of gradient tensor partitioning and multi-level gradient synchronization subtask decomposition;
[0042] Figure 3 This diagram illustrates the dependencies between the forward propagation, backward propagation, and gradient synchronization subtasks.
[0043] Figure 4 The flowchart shows the scheduling process based on priority and communication quota.
[0044] Figure 5 A schematic diagram illustrating the time axis overlap with multi-level communication;
[0045] Figure 6 This is a flowchart of the method of the present invention. Detailed Implementation
[0046] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0047] See Figure 6 The present invention provides a gradient synchronization method for distributed training of hierarchical heterogeneous network topologies, comprising the following steps:
[0048] S1. Obtain the network topology information of the cluster, and identify at least one intra-machine interconnection layer and at least one inter-machine network layer based on the physical connection relationship; in the inter-machine network layer, further identify the intra-rack layer and inter-rack or higher layers based on the rack where the server is located, the switch connection method, etc., thereby forming a hierarchical heterogeneous network topology model containing multiple network layers.
[0049] S2. Based on the hierarchical heterogeneous network topology model, the set of computing nodes participating in data parallel training is grouped according to the network level, and one or more communication groups are divided at each network level; each communication group contains several computing nodes that have high bandwidth or low latency with each other at that level, and the network level to which each communication group belongs and the node identifiers within the group are recorded.
[0050] S3. During the backpropagation process of model training, the gradient tensor of each model layer is split according to the preset partitioning rules to obtain multiple gradient blocks. For each gradient block and each network layer in the hierarchical heterogeneous network topology, the corresponding all-gather gradient synchronization subtask and reduce-scatter gradient synchronization subtask are generated according to the predetermined multi-level communication process.
[0051] S4. Establish reduce-scatter dependency and all-gather dependency for the gradient synchronization subtask, which are passed from low level to high level. At the same time, establish a dependency between the forward computation of each model layer in the next training iteration and the synchronous completion event of the corresponding gradient block.
[0052] S5. Set priority rules for the gradient synchronization subtasks based on model level, communication stage and network level. Organize all gradient synchronization subtasks that have not yet been executed and whose prerequisite conditions have been met into a global priority queue to be scheduled, so that a consistent order of tasks to be scheduled can be obtained on all computing nodes participating in training.
[0053] S6. Set a total communication resource quota for the overall gradient synchronization process and set a hierarchical communication resource quota for each network layer. Based on the global priority queue to be scheduled, select gradient synchronization subtasks that satisfy the preceding dependencies and are within the constraints of the total communication resource quota and the corresponding hierarchical communication resource quota on each computing node participating in training. Initiate reduce-scatter or all-gather communication operations in the corresponding communication group. Dynamically deduct and return the communication resource quota during task execution, thereby controlling the concurrent communication volume on different network layers while ensuring that the link is not overloaded.
[0054] S7. In each training iteration, the aforementioned gradient synchronization subtask scheduling mechanism is embedded into the model training process: During the backpropagation phase, when a model layer completes gradient calculation and generates the corresponding gradient synchronization subtask according to step S3, the subtask is immediately added to the global priority queue for scheduling as described in step S5. The communication resource quota control mechanism described in step S6 continuously selects executable gradient synchronization subtasks from the queue in the background and initiates reduce-scatter or all-gather communication operations to the underlying communication library. When communication is completed, the readiness status of subsequent gradient synchronization subtasks is updated according to the dependencies determined in step S4, so that new executable gradient synchronization subtasks enter the global priority queue for scheduling. Before starting the next round of forward propagation calculation of a model layer, it is determined whether all gradient synchronization subtasks related to that model layer have been completed. If there are incomplete tasks, the start time of the forward calculation of that model layer is postponed until all related gradient synchronization subtasks are completed.
[0055] Example 1
[0056] This embodiment uses a three-layer hierarchical heterogeneous network topology as an example for illustration. Figure 1 As shown, the computing nodes in the cluster are divided into three levels based on their physical connections: the intra-machine interconnection level, the intra-rack network level, and the inter-rack network level. In the intra-machine interconnection level, several graphics processing units within the same server are connected via high-speed interconnect links, forming several intra-machine communication groups. In the intra-rack network level, several servers within the same rack are interconnected via rack switches, forming several intra-rack communication groups. In the inter-rack network level, different racks are interconnected via core switches or higher-level network devices, forming one or more inter-rack communication groups. Each communication group records its network level and the identifiers of the nodes within the group, which are used in subsequent scheduling to determine the target execution range of the gradient synchronization subtasks, corresponding to steps S1 and S2 of the aforementioned method.
[0057] During the backpropagation process of model training, the gradient tensor of each model layer is split according to a preset block division rule. For example, the gradient tensor can be divided into several gradient blocks of roughly the same size according to contiguous memory storage ranges. For each gradient tensor, logically, a full reduction operation on the complete gradient tensor is explicitly decomposed into a reduce-scatter phase and an all-gather phase. Combining the various communication groups within the three-layer network topology (intra-machine, intra-rack, and inter-rack), corresponding gradient synchronization subtasks are generated at multiple network layers (layer 0, layer 1, and layer 2) for the reduce-scatter and all-gather phases, respectively. This results in several reduce-scatter subtasks executed sequentially from lower to higher layers and several all-gather subtasks executed sequentially from higher to lower layers, such as... Figure 2 As shown. This corresponds to step S3 of the aforementioned method.
[0058] While generating gradient synchronization subtasks, the preceding dependencies of each gradient synchronization subtask are explicitly recorded. For example, consider a two-layer model, each layer containing two gradient blocks. Figure 3 As shown, the gradient of model layer 0 is divided into blocks g. 00 g 01 The gradient of model layer 1 is divided into blocks g. 10 g 11 For the same gradient block g ij The protocol dissemination subtask RS_L0(g) belongs to the intra-machine interconnect layer. ij The backpropagation calculation of event B depends on the corresponding model layer. i ; This refers to the protocol dissemination subtask RS_L1(g) at the rack-mounted network layer. ij Depends on RS_L0(g) ij ) Completion event; RS_L2(g) is a protocol dissemination subtask belonging to the inter-rack network layer. ij Depends on RS_L1(g) ij The completion of the event forms a protocol dependency link that propagates from lower to higher levels. The aggregation collection subtask AG_L2(g) belongs to the inter-rack network layer. ij Depends on RS_L2(g) ij ) Completed event; belongs to the aggregate collection subtask AG_L1(g) at the rack network level. ij ) depends on AG_L2(g ij Completion event; Aggregation collection subtask AG_L0(g) belongs to the intra-machine interconnection level. ij Depends on AG_L1(g) ijThe completion of the event forms a collection dependency chain that propagates from higher to lower levels. Furthermore, the corresponding model layer forward propagation operator F... i The launch event is set to depend on the aggregation collection subtask AG_L0(g) of all gradient blocks in the model layer at the intra-machine interconnect level. ij This completes the event, thereby establishing a constraint relationship between the gradient synchronization subtask and the forward propagation computation at the dependency level. This corresponds to step S4 of the aforementioned method.
[0059] Regarding scheduling priority rules, this embodiment assigns attributes such as the model layer number L, communication phase identifier P, target network layer sequence number H, and block sequence number C to each gradient synchronization subtask. The communication phase identifier P distinguishes between the reduction propagation phase and the aggregation collection phase, with 0 for the reduction propagation phase and 1 for the aggregation collection phase. The target network layer sequence number H is numbered sequentially from highest to lowest network layer, with smaller numbers representing higher network layers. Based on these attributes, a priority vector (L, P, H, C) is constructed for each gradient synchronization subtask. When comparing the priorities of two gradient synchronization subtasks, the priority is first determined by L from smallest to largest. If L is the same, the priority is determined by P from smallest to largest. If both L and P are the same, the priority is determined by H from smallest to largest. Finally, if all three components are the same, the priority is determined by the block sequence number C from smallest to largest. By using the priority rules described above, a consistent global priority queue for scheduling can be built on all computing nodes participating in training. Each gradient synchronization subtask taken from the head of the queue is the task with the highest current priority and whose preceding dependencies have been satisfied, corresponding to step S5 of the aforementioned method.
[0060] In terms of communication resource management and scheduling, such as Figure 4As shown, this embodiment uses the amount of transmitted data as the unit of measurement, sets a total communication resource quota T_credit for the overall gradient synchronization process, and sets hierarchical communication resource quotas L_credit[0], L_credit[1], and L_credit[2] for the three levels of intra-machine interconnect, intra-rack network, and inter-rack network, respectively, to represent the maximum amount of data allowed to be transmitted simultaneously within the corresponding range. During the training process, each computing node participating in the training maintains a global priority queue that is consistent with the aforementioned priority rules. The global priority queue only contains gradient synchronization subtasks whose preceding dependencies have been satisfied but have not yet been executed. During scheduling, the following process is executed cyclically: First, the highest priority gradient synchronization subtask t is retrieved from the head of the priority queue. The data size size(t) and target network level level(t) to be transmitted for this task are obtained. It is then determined whether size(t) ≤ T_credit and size(t) ≤ L_credit[level(t)] are satisfied simultaneously. If both are satisfied, a reduction broadcast or aggregation collection communication operation is initiated within the corresponding communication group. The task is marked as executing, and size(t) is deducted from T_credit and the corresponding level's L_credit[level(t)]. If the current task t does not meet the communication resource quota constraint, the process continues to check whether there are other tasks to be scheduled in the priority queue. If so, the next candidate task t' is selected according to priority order. The above quota determination process is repeated until a task that meets the quota constraint is found in the queue and communication is initiated, or all tasks in the queue cannot be initiated temporarily due to insufficient quota. When all tasks in the priority queue do not meet the communication resource quota constraint, the initiation of new gradient synchronization subtasks is temporarily stopped, and the existing gradient synchronization subtasks are waited for completion and release of communication resource quota. When the underlying communication library reports on each computing node that a certain gradient synchronization subtask t has been completed, the corresponding value is added back to T_credit and L_credit[level(t)] according to the actual data size size(t) of the task, and the status of the task is updated to completed. For subsequent gradient synchronization subtasks that depend on t, their dependency counts are updated. When all the preceding dependencies of the gradient synchronization subtask have been completed, it is inserted into the global priority queue to wait for subsequent scheduling. The above process corresponds to step S6 of the aforementioned method.
[0061] During the training iteration process, this embodiment drives multi-level communication and computation to operate collaboratively according to the gradient synchronization subtask generation and scheduling mechanism defined in steps S3 to S6. On the one hand, each computing node within the same communication group calls the corresponding reduction propagation and aggregation collection communication operations in a consistent order through a global priority queue for scheduling and unified dependency management rules, thereby avoiding deadlocks or inconsistent results caused by inconsistent communication call order. On the other hand, under the control of communication resource quota constraints, gradient synchronization subtasks from different model layers and different gradient blocks are interleaved at multiple network layers, so that different blocks of reduction and collection operations are carried on higher-level and lower-level network links respectively, forming a pipeline-like overlapping relationship in time. A possible scheduling situation is as follows: Figure 5 As shown, after the backpropagation calculation B1 is completed, the protocol dissemination task RS on the internal Level 0 communication line... 10 RS 11 Add to the global priority queue for scheduling, RS 10 Execution is immediate. Once the backpropagation calculation of B0 is complete, the protocol dissemination task RS on the internal Level 0 communication line is executed. 00 RS 01 Add to the global priority queue for scheduling, at which point RS 10 It has been completed, and because of RS 00 RS 01 Priority greater than RS 11 Therefore, it receives priority for scheduling and execution. When the reduction dissemination task at a certain level is completed, the reduction dissemination task of the same block at a higher level is added to the queue, so that the reduction operation of that block proceeds step by step along the network hierarchy, while other blocks continue to be reduced at lower levels, thus forming a staggered and superimposed communication pipeline between network layers. When the reduction dissemination task RS at the highest level (Level 2) is completed... 00 RS 01 Upon completion, the corresponding aggregation and collection task AG 00 AG 01 Once the dependencies are satisfied, the process can begin at Level 2 and proceed step-by-step to Level 1 and Level 0, interleaving execution with other blocks that have not yet completed their reductions. For each model layer's forward propagation, before its forward operator is scheduled for execution, it must be checked whether all gradient synchronization subtasks corresponding to each gradient block associated with that model layer have been completed. If any subtasks are incomplete, the forward propagation operator at that model layer will wait for the corresponding gradient synchronization subtasks to complete before executing the forward computation. Forward computation F0 waits for the aggregation collection task AG at Level 0. 00 AG 01Once completed, the process can begin. Simultaneously, the aggregation and collection tasks that the forward computation F1 depends on continue to execute on Level 1 and Level 2. After their completion, the forward computation F1 will then begin. This approach allows gradient synchronization subtasks of other model layers to continue executing in the background, ensuring that the parameters of the current model layer have been synchronously updated. This achieves temporal overlap between multi-level communication and forward and backward propagation, corresponding to step S7 of the aforementioned method.
Claims
1. A gradient synchronization method for distributed training in hierarchical heterogeneous network topologies, characterized in that, The method includes: Obtain the network topology information of the distributed training cluster, and construct a hierarchical communication topology model containing multiple network layers based on the physical connection relationship between training nodes; The gradient tensors of each layer of the neural network model to be trained are split into multiple gradient blocks according to a preset rule; Based on the hierarchical communication topology model, the gradient block synchronization operation is divided into multiple gradient synchronization subtasks; According to a preset scheduling strategy, the gradient synchronization subtask is scheduled to a communication library pre-configured on the training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster. The gradient blocks in other training nodes of the distributed training cluster are obtained through the communication library.
2. The gradient synchronization method in distributed training as described in claim 1, characterized in that, The construction of a hierarchical communication topology model containing multiple network layers based on the physical connection relationships between training nodes includes: Identify at least one intra-machine interconnection layer and at least one inter-machine network layer based on physical connectivity relationships; In the inter-machine network hierarchy, the rack-level, inter-rack level, or higher level are further identified based on the rack and switch connection method of the training node.
3. The gradient synchronization method in distributed training as described in claim 1, characterized in that, The gradient tensors of each layer of the neural network model to be trained are split into multiple gradient blocks according to a preset rule, including: The gradient tensor is divided into several gradient blocks according to the continuous storage address range of the gradient tensor in the memory, and the amount of data corresponding to each gradient block is the same within a preset range.
4. The gradient synchronization method in distributed training as described in claim 1, characterized in that, Based on the hierarchical communication topology model, the gradient block synchronization operation is divided into multiple gradient synchronization subtasks, including: A gradient synchronization operation is logically decomposed into a reduction propagation phase and an aggregation collection phase. In the hierarchical communication topology model, corresponding reduction dissemination subtasks and aggregation collection subtasks are generated at each network layer, so that each gradient synchronization subtask corresponds to at least a gradient block, a network layer, and a gradient synchronization stage.
5. The gradient synchronization method in distributed training as described in claim 1, characterized in that, The step of scheduling the gradient synchronization subtask to the communication library pre-configured on the training node according to the preset scheduling strategy includes: Based on the hierarchical communication topology model, inter-task dependencies are established for the gradient synchronization subtasks; a preset priority rule is set for the gradient synchronization subtasks, and all gradient synchronization subtasks that have not yet been executed and whose preceding dependency conditions have been met are organized into a global priority queue to be scheduled according to the priority rule. Set the total communication resource quota, and set the hierarchical communication resource quota for each network layer of the hierarchical communication topology model; Based on the global priority queue to be scheduled, gradient synchronization subtasks that meet the constraints of the total communication resource quota and the corresponding level communication resource quota are selected cyclically from the global priority queue to be scheduled on each training node participating in the training. These subtasks are then submitted to the communication library pre-configured in the training node, and the communication resource quota is dynamically deducted and returned during task execution.
6. The gradient synchronization method in distributed training as described in claim 5, characterized in that, The step of establishing inter-task dependencies for the gradient synchronization subtasks based on the hierarchical communication topology model includes: For the same gradient block, the completion of a reduction dissemination subtask on a certain level network is a prerequisite for the reduction dissemination subtask on a higher level network. For the same gradient block, the completion of the aggregation collection subtask on a certain level network is a prerequisite for the aggregation collection subtask on the next lower level network. For the same gradient block, the completion of the reduction dissemination subtask at the highest network level is a prerequisite for the aggregation collection subtask at the highest network level. The reduction dissemination subtask at the lowest network level has no preconditions.
7. The gradient synchronization method in distributed training as described in claim 5, characterized in that, The step of setting a preset priority rule for the gradient synchronization subtask, wherein the priority rule determines the scheduling priority of the gradient synchronization subtask based on at least the following information: The neural network model level to which the gradient synchronization subtask belongs is used to give gradient synchronization subtasks in earlier model layers a higher priority than those in later model layers. The communication phase to which the gradient synchronization subtask belongs is used to give priority to gradient synchronization subtasks in the reduction and dissemination phase over gradient synchronization subtasks in the aggregation and collection phase. The communication topology network hierarchy of gradient synchronization subtasks is used to give priority to gradient synchronization subtasks located at higher network levels than those located at lower network levels.
8. The gradient synchronization method in distributed training as described in claim 1, characterized in that, After obtaining gradient blocks from other training nodes in the distributed training cluster through the communication library, the method further includes: After all the gradient blocks belonging to the same gradient tensor have been acquired, they are reassembled into a complete gradient tensor. Update the parameters of the corresponding tensor in the neural network model to be trained in the current training node based on the gradient tensor.