Communication scheduling method based on hybrid expert parallel mechanism

Abstract

The invention belongs to the technical field of large-scale deep learning model training, and particularly relates to a communication scheduling method based on a hybrid expert parallel mechanism. The communication scheduling method comprises three parts of constructing a load balancing module, a communication resource allocation module and a parallelization scheduling module, and according to the popularity degree of an expert model and the load calculation condition, parameter slices and mapping strategies of the expert model are dynamically adjusted to realize load balancing; and changing a primitive execution mode of the communication library, enabling All-Reduce operation to support asynchronous execution, and processing All-to-All and All-Reduce communication tasks in parallel through a multi-thread mechanism. And load balancing and parallelization of All-to-All communication between the nodes and All-Reduce communication in the nodes are realized in a model training process. According to the method, the link bandwidth is fully utilized while the data consistency is maintained, and the communication waiting time is reduced. Experimental results show that the GPU utilization rate is effectively improved in hybrid expert model training, and the training time is shortened.

Description

Technical Field

[0001] The present invention belongs to the technical field of large-scale deep learning model training, and particularly relates to a communication scheduling method based on a mixture of experts parallel mechanism. Background Art

[0002] Sparsely activated models are a strategy proposed to address the high costs of large dense models in terms of computation and memory requirements [1]. In traditional dense models, all parameters are activated in each computation. Although this approach is simple and straightforward, it leads to huge computational overhead and resource consumption in large-scale model and data training. As the scale of deep learning models continues to expand, this computational cost and memory requirement have gradually become bottlenecks in model training. Therefore, researchers have proposed sparsely activated models, which only activate a part of the neural network units in each computation, thereby reducing the amount of computation, improving computational efficiency, and significantly reducing resource consumption while maintaining the model's expressive power. Mixture of Experts (MoE) is a typical implementation of sparsely activated models [2][3]. The MoE model divides the entire network into multiple independent expert components and a gating network. Among them, an "expert" or "expert model" refers to the sub-network module that is selectively activated in the MoE structure, usually a set of neural network modules with the same structure but independent parameters. Each expert can be regarded as a sub-model with a specific function; the "gating network" is a control module used to determine which expert models to activate based on the input features, usually deciding the subset of experts participating in the computation through a sparse selection mechanism (such as Top-k). The gating network is responsible for selecting and activating a small number of experts based on the input data, thereby significantly reducing the computational overhead while ensuring the model's expressive power. This design enables the MoE model to maintain the expansion of the model scale while the computational complexity only grows linearly with the number of activated experts. The MoE model not only demonstrates powerful performance advantages in the fields of natural language processing, computer vision, etc., but also provides new ideas for large-scale model training.

[0003] In large-scale MoE model training, the increase in the number of experts leads to the inability of a single GPU memory to accommodate all experts, so an expert parallel strategy has been proposed [4]. This strategy distributes different experts on different GPUs and performs computations in parallel, thereby breaking through the memory limit of a single GPU and making full use of distributed computing resources. "Expert placement" refers to the process of mapping each expert model to a specific physical GPU or computing node, and its goal is to achieve an optimal trade-off among computing resources, communication bandwidth, and load balancing. However, this also introduces new problems, such as the complexity of cross-GPU communication, the instability of training efficiency, and the imbalance of resource utilization. As Figure 1As shown in the figure, two-way data parallelism with four expert models is adopted, where two expert models are deployed on each node. Since the experts are distributed on different worker nodes, All-to-All communication is required in both the front and back stages of expert parallelism to allocate input tokens and return the calculation results of the expert models. Although the expert parallelism strategy solves the memory bottleneck, it also exposes various technical challenges in distributed training, which directly affect the performance and stability of the MoE model in large-scale training.

[0004] The MoE model faces two major challenges during training:

[0005] Communication overhead: The expert parallelism strategy requires frequent All-to-All communication between different GPUs, resulting in insufficient utilization of computing resources. In addition, although there is no direct competition for resources between the All-to-All communication between nodes and the intra-node All-Reduce communication caused by tensor parallelism among experts, due to their dependence in the execution order, the communication link fails to be fully utilized, causing a certain degree of resource waste. At the same time, during the backpropagation of the model, resource competition occurs between the All-to-All communication of the MoE layer and the gradient All-Reduce operations of other layers, resulting in a decrease in the overall communication efficiency. As the model scale and the number of experts increase, the communication overhead increases exponentially, becoming the main bottleneck for training speed and scalability.

[0006] Load imbalance: Due to the preference of the gating network in selecting experts, some experts will be over-activated, resulting in uneven distribution of computing and communication loads. This not only reduces the training efficiency of the model but also affects the stability of training. The load imbalance problem will further exacerbate the communication overhead, severely affecting the latency and throughput during the training process. Summary of the Invention

[0007] To solve the problems of insufficient utilization of computing resources, low communication efficiency, uneven load between nodes, and inability to adapt to dynamic resource changes in the training of the current Mixture of Experts (MoE) model, the present invention proposes a communication scheduling method based on a hybrid expert parallel mechanism.

[0008] The communication scheduling method based on the hybrid expert parallel mechanism provided by the present invention includes three parts: a load balancing module, a communication resource allocation module, and a parallelization scheduling module. The specific steps are as follows:

[0009] Step S1: Design the load balancing module, that is, construct the expert model allocation strategy: Dynamically generate the parameter slicing and mapping strategy of the expert model according to the popularity, computing load of the expert model, and the resource status of the GPU cluster, so as to achieve the efficient coordination and balanced utilization of computing and communication resources. Specifically, an intelligent allocation mechanism is adopted to preferentially map the expert models with high computing frequency and large communication requirements to multiple nodes in the form of parameter slicing, so as to provide more computing and communication resources for these expert models. Such a design is based on the strong correlation between the popularity of the expert model and the computing resource requirements: The expert models with high popularity are often called more frequently and have greater computing requirements. Among them, "popularity" refers to the frequency of being selected by the gating network during the training process, that is, the probability that the expert is activated to participate in the calculation in all input samples is higher. If the number of allocations of all expert models on the GPU is exactly the same, or the types of GPU resources used by popular and unpopular expert models in the heterogeneous cluster are the same, it will lead to an imbalance in resource utilization efficiency. The popular expert models may become performance bottlenecks due to insufficient resources, while the unpopular expert models may cause computing resources to be idle. Therefore, through the dynamic slicing and mapping strategy, the number and type of GPUs can be reasonably allocated to each model according to the actual computing and communication requirements of the expert model, not only achieving the balanced distribution of computing load and communication requirements, but also maximizing the utilization efficiency of the overall resources of the cluster.

[0010] Step S2: Design a communication resource allocation module (model). Since expert parallelism requires a large amount of All-to-All communication, different expert placement strategies will involve different combinations of inter-machine communication and intra-machine communication. In view of this characteristic, a link resource allocation model for cross-node and intra-node communication is designed, and a multi-hop transmission strategy is introduced to achieve efficient and reasonable allocation of communication link resources. The so-called "expert placement strategy" refers to the specific scheme of mapping each expert model to which GPUs or computing nodes. The placement location of the expert model determines the complexity of the communication path. For example, when all experts are assigned to the same computing node (such as a server with 8 GPUs), all communications can be completed through high-speed intra-node interconnection (such as NVLink), only involving intra-machine communication; while when the expert models are distributed on multiple computing nodes, that is, there is a case of cross-server placement, then All-to-All communication will involve both inter-machine communication (through InfiniBand, RDMA, etc.) and intra-machine communication. In this case, the communication structure is more complex and the bandwidth bottleneck is more prominent. In specific implementation, the All-to-All communication forms under different expert placement strategies may vary: some only involve inter-machine communication, and some involve both inter-machine and intra-machine communication. For the latter complex communication scenario, a multi-hop transmission strategy is adopted for optimization. Specifically, when All-to-All communication spans both inter-machine and intra-machine links at the same time, the data that needs to be transmitted to multiple GPUs within the same node can be first centrally sent to a certain GPU on this node, and then this GPU distributes the data to other GPUs within this node. This multi-hop transmission method can effectively reduce the scale of inter-machine communication and the cross-node bandwidth requirements, thereby improving the overall bandwidth utilization rate of All-to-All communication. At the same time, the communication speed based on NVLink within the node is much higher than the network communication speed across nodes, which can further accelerate the data transmission efficiency and give full play to the performance advantages of the high-speed interconnection architecture within the cluster.

[0011] Step S3: Design a parallelized scheduling module. This module aims to efficiently coordinate the All-to-All communication between nodes and the All-Reduce communication within nodes, maximize the utilization of communication bandwidth, and reduce the overall communication latency. Specifically, an asynchronous executor is introduced to enable the All-Reduce operation to support asynchronous execution, thereby enhancing the parallelism of communication. The asynchronous executor is a communication control component that runs at the scheduling layer. By modifying the execution mode of the primitives of the underlying communication library (such as NCCL or Gloo), it converts the originally blocking communication operation into a non-blocking execution, allowing multiple communication tasks to be scheduled concurrently. The executor creates an asynchronous context for each communication operation and tracks its completion status through an event listening mechanism to avoid resource idling. This mechanism can refer to the asynchronous primitive execution mechanism in the NVIDIA NCCL library [5] or the communication asynchronous optimization strategy in Megatron-LM [6] (Note: The NCCL library supports asynchronous communication primitives, such as non-blocking All-Reduce operations, which can be used as the basis for the underlying implementation of the asynchronous executor; in Megatron-LM, the communication-computation overlap (such as the asynchronous execution of gradient All-Reduce) is used to optimize the communication scheduling strategy for large-scale model training). At the same time, a multi-threaded mechanism is adopted to perform chunking processing on the data that requires both inter-machine All-to-All and intra-machine All-Reduce communication. The so-called "data chunking" is to split the tensor participating in multiple communication tasks along a certain dimension into several sub-chunks, and each sub-chunk performs communication operations independently. In this way, the originally serial communication process can be executed in parallel. For example, while a part of the data chunks are performing All-to-All communication, another part of the data chunks can perform All-Reduce operations in parallel. The chunking strategy not only improves the bandwidth utilization rate but also enhances the pipelining execution ability of communication operations. The choice of chunking granularity needs to be dynamically optimized in combination with communication overhead, network topology, and GPU memory capacity. In this way, while a part of the data is performing inter-machine All-to-All communication, another part of the data is performing intra-machine All-Reduce communication, fully exploiting the concurrent capabilities of the network link. On the premise of ensuring data consistency, this module effectively reduces the communication waiting time and significantly improves the data transmission rate.

[0012] Furthermore:

[0013] In step S1, the specific process of constructing the expert model allocation strategy is as follows: First, define which expert models each GPU should be responsible for executing to achieve load balancing of the expert models on the GPUs. According to the given model structure and GPU resources, an optimization model is established with the goal of maximizing the training efficiency. This modeling process aims to determine "which experts should be placed on which GPUs" to balance the computational pressure and communication requirements. Different placement strategies will significantly affect the operating efficiency of the system: If multiple frequently activated experts are concentrated on a few GPUs, although the computing is concentrated, it will cause bottlenecks in these GPUs and lead to the problem of excessive computing load; while if the experts are distributed among multiple nodes, cross-node communication needs to be frequently executed in each round of training, significantly increasing the All-to-All bandwidth pressure and communication latency. Therefore, the distribution of the expert models not only determines the computational tasks borne by each GPU but also directly affects the amount of data exchanged between GPUs. By constructing an optimization model, an optimal expert placement plan can be found based on factors such as the computing power of the GPUs, the bandwidth limit, and the expert activation frequency, ultimately achieving the goal of maximizing the training efficiency. The placement strategy modeling of the expert models is as follows:

[0014] Taking a single layer containing E expert models as an example, let the number of nodes for placing the expert models be W, the total number of GPUs be N, each node is equipped with C homogeneous GPUs, and use F i to represent the computing power of the GPU where expert i is located, and use BW m,n to represent the communication bandwidth from GPUm to GPUn, and the expert tensor parallelism is T. For the i-th expert model, let x i,n ∈{0,1} indicate whether it is placed on the n-th GPU, and y i,w ∈{0,1} indicate whether it is placed on the w-th host. For simplicity, assume that each input token only selects one expert model for training. According to the output of the gating network (since MoE adopts a sparse activation strategy, the gating network determines which experts to activate), the amount of data that the i-th expert needs to transmit to the j-th expert can be obtained as D i,j .

[0015] Constraints:

[0016] (a) Each expert can only be placed within the same physical node, that is, the constraint condition for the i-th expert to be placed on the same physical node is

[0017] (b) To ensure that the computing resources of each node are not overloaded, define the maximum number of GPUs that can be allocated to each node to avoid over-allocation in the case of limited resources. That is, the constraint on the number of GPUs that can be allocated on the w-th worker node is:

[0018]

[0019] Among them, is used to restrict that the \(i\)-th expert is placed in a node and the \(j\)-th expert is placed in a node (i.e., placed in the same node);

[0020] Specific explanation: Since the scale of the current expert model parameters is relatively large, a single GPU often cannot fully accommodate a complete expert model. Therefore, the system generally adopts the Tensor Parallelism strategy, that is, the parameters of an expert are sliced into multiple sub-tensors and mapped to multiple GPUs respectively for parallel computing [7]. Although this slicing improves scalability in terms of computing resources, it also brings an important problem: if these GPUs used for tensor parallelism are distributed on different physical nodes, then after each forward or backward propagation ends, it is necessary to aggregate the sliced calculation results through cross-node All-Reduce communication, which will significantly increase the communication latency and bandwidth pressure. On the contrary, if these GPUs are all located on the same physical node, the aggregation operation can be completed through high-speed intra-node interconnection (such as NVLink), and the communication efficiency is greatly improved. Therefore, in order to reduce the cross-node communication overhead, it is necessary to add a restrictive condition during the placement of the expert model so that the sub-blocks sliced by tensor parallelism are distributed in the same node as much as possible to ensure that its aggregation operation is completed within the node.

[0021] 1) One expert is located on one GPU:

[0022] Communication duration from the \(i\)-th expert to the \(j\)-th expert:

[0023]

[0024] Communication duration of the \(i\)-th expert:

[0025]

[0026] 2) One expert is located on multiple GPUs:

[0027] [[ID=2,9]]Communication duration of the \(i\)-th expert:

[0028]

[0029] Computation duration of the \(i\)-th expert:

[0030]

[0031] Optimization problem:

[0032]

[0033] minimize \(T\),

[0034] st(a)(b), where T i comp represents the computation time of the i-th expert; 2T i comp represents the communication duration of the i-th expert.

[0035] In step S2, the communication link resource allocation model is designed, specifically:

[0036] Define the communication path from the i-th expert to the j-th expert That is, the communication order is iklj, and the corresponding bandwidth is Scale Factor The data transmitted on this path accounts for the total data D i,j proportion.

[0037] Restrictions:

[0038] c) Ensure that the total amount of data transmitted on each communication path is equal to the amount of data that needs to be transmitted between expert models, that is, the total amount of data transmitted in all communication paths from the i-th expert to the j-th expert is equal to D i,j ,Right now

[0039] d) Ensure that when two expert models are in the same node, their communication paths are only carried out within the node to avoid cross-node communication delays, that is, the i-th expert and the k-th expert are in the same node; the j-th expert and the l-th expert are in the same node, that is, y i,w =y k,w ,y j,w =y l,w ;

[0040] e) Ensure that the expert models in different nodes can communicate synchronously, that is, the i-th and k-th experts are located in the same node, and the l-th and j-th experts are located in the same node, that is,

[0041] in, Used to restrict the i-th expert to be placed in a node and the j-th expert to be placed in a node:

[0042] 1) One expert is located on one GPU:

[0043] Communication time from the i-th expert to the j-th expert:

[0044]

[0045] The communication duration of the i-th expert:

[0046]

[0047] In formula (6), the first term is the transmission time of the data of the \(i\)-th expert \(D\) i,j , the second term is the time for transmitting data when at the second node in the path, and the third term is the time for transmitting data when at the third node in the path; \(\beta\) represents the synchronization condition where the \(i\)-th expert and the \(k\)-th expert are in the same node, and the \(l\)-th expert and the \(j\)-th expert are in the same node. When \(\beta = 1\), the constraint of synchronous communication among experts within the above node is satisfied; otherwise \(\beta = 0\). \(\beta_1\) and \(\beta_2\) respectively represent the inter-node synchronization conditions in the second-hop and third-hop stages of the communication path. For example, \(\beta_1 = 1\) means that the experts at the second node in the path satisfy the synchronous communication constraint, and \(\beta_2 = 1\) means that the experts at the third node in the path satisfy the synchronous communication constraint.

[0048] 2) One expert is located on multiple GPUs:

[0049] Communication duration from the \(i\)-th expert to the \(j\)-th expert:

[0050]

[0051] Communication duration of the \(i\)-th expert:

[0052]

[0053] Computation duration of the \(i\)-th expert:

[0054]

[0055] Problem to be optimized:

[0056]

[0057] where \(T\) i comp represents the computation duration of the \(i\)-th expert; \(2T\) i comm represents the communication duration of the \(i\)-th expert. The definitions of \(\beta\), \(\beta_1\) and \(\beta_2\) are the same as in formula (6), while \(\gamma\) represents the cross-node synchronization constraint where the \(i\)-th expert and the \(j\)-th expert are in different nodes. When \(\gamma = 1\), the \(i\)-th expert and the \(j\)-th expert are respectively in different nodes, and the communication path meets the cross-node synchronization requirements; otherwise \(\gamma = 0\). \(\gamma_1\) and \(\gamma_2\) respectively represent the synchronization conditions of cross-node communication in different stages of the communication path. For example, \(\gamma_1 = 1\) means that the cross-node communication in the second hop of the path satisfies the synchronization constraint, and \(\gamma_2 = 1\) means that the cross-node communication in the third hop of the path satisfies the synchronization constraint.

[0058] In step S3, through the expert placement strategy and the communication link resource allocation strategy, not only is the load balancing of resources achieved, but also the network topology structure is fully considered to achieve efficient communication. And a parallel scheduling module is designed to hide the high-overhead All-to-All communication. In the current large model training, the expert model usually adopts the tensor parallelism method, distributing the parameters of each expert on multiple GPUs. Although this method can effectively utilize hardware resources, it also brings some communication bottlenecks. Take Figure 2 (a) as an example. After the input tokens pass through the attention module and the gating network, first, the All-to-All communication between machines is performed to transfer the data to the node where the target expert is located. Then, after the calculation of the expert layer, the tensors need to be aggregated through the All-Reduce communication within the node. Finally, the calculation results are transmitted back to the node where the tokens were originally located through the All-to-All communication between machines again. In the existing scheme, the All-to-All communication between machines and the All-Reduce communication within the machine are interleaved. Specifically:

[0059] 1) All-to-All communication between machines: It is used to transfer data between different nodes and is usually carried out through the RDMA link.

[0060] 2) All-Reduce communication within the machine: It is used to aggregate tensors between multiple GPUs within the node and uses in-machine links such as NVLink.

[0061] Since these two communication methods use different link resources (All-Reduce uses in-machine NVLink, and All-to-All uses the inter-machine RDMA link), they are alternately carried out in sequence in the existing scheme. This alternating communication method may lead to underutilization of link resources. For example, when performing All-to-All communication, the in-machine link may be idle; similarly, when performing All-Reduce communication, the inter-machine link may be idle, resulting in waste of communication resources. To solve the above problem of insufficient utilization of communication resources, a key point of optimization is to parallelize the All-to-All communication between machines and the All-Reduce communication within the machine. The specific method is:

[0062] (1) Data splitting: Split the tensors that need to perform All-Reduce and All-to-All operations simultaneously. For example, split half of the data for All-Reduce operation, then perform All-to-All communication on this part of the data; at the same time, perform All-to-All communication on the other half of the data, and then perform All-Reduce operation on this part of the data. In this way, the overlapping use of resources can be achieved during the two communication operations.

[0063] (2) Parallel scheduling: Through a reasonable scheduling mechanism, ensure that the two communication operations can be carried out in parallel, so as to maximize the utilization of link resources and reduce communication latency.

[0064] This optimization strategy can make full use of the inter-machine and intra-machine communication links by parallelizing communication operations at different levels, reduce the overall communication execution time, and thus improve the overall training efficiency. To verify the effectiveness of this optimization scheme, the present invention is tested through theoretical analysis and small-scale experiments (in an experimental environment with two machines and eight cards). The results show that hiding the All-Reduce communication of the expert layer under the All-to-All communication can improve the communication performance by about 12%, and the overall training performance is improved by 5%.

[0065] In the specific implementation, an asynchronous executor is also introduced. This executor adjusts the primitive execution mode of the underlying communication library to support the asynchronous execution of key communication operations such as All-Reduce. At the same time, an efficient thread pool is introduced to manage the execution of parallel tasks. Each thread in the thread pool is dedicated to handling an independent communication task, such as data exchange in inter-machine All-to-All communication or data aggregation in intra-machine All-Reduce communication.

[0066] In terms of performance evaluation, the PyTorch Profiler tool is used to monitor and compare the execution time of a single communication kernel in the model training process in detail. The experimental results show that with the help of the asynchronous executor, the basic overlap of inter-machine All-to-All communication and intra-machine All-Reduce communication is successfully achieved.

[0067] Based on the above communication scheduling method, the present invention also includes a communication scheduling system based on the hybrid expert parallel mechanism. This communication scheduling system includes the above-mentioned load balancing module, communication resource allocation module, and parallel scheduling module.

[0068] The present invention realizes the following functions:

[0069] The present invention addresses the problems of resource utilization, communication efficiency, load balancing, and dynamic resource adaptation in the training of the mixture-of-experts model, and proposes a complete and efficient communication scheduling system, which greatly improves the performance and scalability of large model training.

[0070] (1) Efficient resource allocation and load balancing: Dynamically generate the parameter slicing and mapping strategies of the expert model, and based on the popularity of the expert model, the computational load, and the resource status of the GPU cluster, achieve the intelligent allocation of computing and communication resources. Prioritize mapping the expert models with frequent computations and large communication requirements to multiple nodes to ensure load balancing and efficient resource utilization.

[0071] (2) Communication link optimization: For cross-node and intra-node communications, design a communication link resource allocation model, and through the multi-hop transmission strategy, achieve the reasonable allocation of communication link resources. Ensure the efficient cooperation of cross-node communication and intra-node communication in data transmission, and reduce communication bottlenecks.

[0072] (3) Parallelization of communication scheduling: Change the execution mode of the primitive of the traditional communication library to enable the All-Reduce operation to support asynchronous execution. Coordinate the cross-node All-to-All communication and the intra-node All-Reduce communication to achieve the parallel execution of these two types of communication operations, thereby reducing the overall communication time and accelerating the training process.

[0073] (4) Optimize the placement of the expert model and the communication duration: By establishing an optimization model, maximize the training efficiency according to the GPU computing power and communication bandwidth. Balance the computing and communication durations of the expert model, reduce the cross-node data transmission delay, and optimize the training process.

[0074] Experimental results show that in the training of the mixture-of-experts model, the GPU utilization rate is effectively improved and the training time is reduced. Description of the Drawings

[0075] Figure 1 It is the mixture-of-experts parallel mechanism in model training.

[0076] Figure 2 It is the design of the communication scheduler scheme.

[0077] Figure 3 It is the architecture of the communication scheduling system based on the mixture-of-experts parallel mechanism. Detailed Implementation Manner

[0078] The communication scheduling method based on the mixture-of-experts parallel mechanism, the specific steps are as follows:

[0079] Step 1: The expert model allocator dynamically generates parameter slices and mapping strategies for expert models based on the popularity of the expert models, the computational load, and the resource status of the GPU cluster. An optimization model is established with the goal of maximizing the training efficiency and achieving load balancing of the expert models on the GPUs. The model gives priority to mapping expert models with high computational frequency and large communication requirements to multiple nodes. The constraints include: (a) Each expert can only be placed within the same physical node; (b) The GPU resources of each node cannot be overloaded. Finally, load balancing and efficient utilization of resources are achieved through an intelligent allocation mechanism.

[0080] Step 2: Communication link resource allocation. The communication of expert models involves cross-node and intra-node GPU communication. To effectively manage communication resources, a communication link resource allocation model is designed. The model defines the bandwidth and data ratio of each communication path to ensure that the data volume on the communication path is consistent with the requirements between expert models. The constraints include: (c) The total amount of data transmitted on the communication path is equal to the amount of data that needs to be transmitted between models; (d) Expert models within the same node only perform intra-node communication; (e) Cross-node communication needs to ensure synchronous execution. Through multi-hop transmission strategies and path optimization, communication latency is effectively reduced.

[0081] Step 3: Bandwidth optimization. Based on a topology-aware method, cross-node All-to-All communication and intra-node All-Reduce communication operations are reasonably arranged. Topology information is used to reduce link congestion, and multi-hop transmission strategies are used to improve communication efficiency. At the same time, through a hierarchical communication strategy, inter-machine communication and intra-machine communication are executed in parallel.

[0082] Step 4: Task distribution. A multi-threaded mechanism is adopted to make All-to-All and All-Reduce communication tasks execute in parallel. Multi-threaded scheduling improves the utilization rate of the network link bandwidth and significantly reduces communication latency. Combined with the bandwidth optimization strategy, the maximum overlap of computing and communication operations is achieved.

[0083] Step 5: Asynchronous execution. The execution mode of the original primitives of the traditional communication library is changed to make the All-Reduce operation support asynchronous execution. Through the asynchronous execution mechanism, a high degree of overlap between intra-machine communication and inter-machine communication is achieved, reducing the overall communication time.

[0084] Experimental verification. To verify the effectiveness of the communication scheduling method based on the hybrid expert parallel mechanism proposed in the present invention, a series of small-scale empirical experiments were designed and carried out. The experiments were conducted in an environment consisting of 2 servers, each configured with 4 GPUs (a total of 8 NVIDIA A100 GPUs, interconnected by NVLink, and using InfiniBand to support cross-machine communication). The deep learning model adopted was a typical Mixture of Experts (MoE) architecture, the model training framework was the Megatron-DeepSpeed framework, the communication operations were implemented through the NCCL library, and its execution mode was customized and optimized to support asynchronous communication.

[0085] (1) Experimental settings:

[0086] Model structure: An MoE model with multiple expert layers was adopted. Each layer contained 8 experts, and 2 experts were activated in each forward pass (Top-2 routing).

[0087] Training task: The C4 dataset subset was used for the pre-training task, and the batch size was set to 32.

[0088] Parallel strategy: A hybrid parallel strategy combining tensor parallelism and expert parallelism was adopted. The tensor parallelism granularity was 4 cards per machine. The expert layers completed data distribution through All-to-All communication and parameter aggregation through All-Reduce.

[0089] Baseline scheme: The traditional synchronous communication mechanism was used, and the All-Reduce and All-to-All operations were executed serially in order.

[0090] Optimized scheme: The communication scheduling method proposed in the present invention was introduced, using an asynchronous executor, thread pool scheduling, multi-hop path optimization, and topology-aware scheduling.

[0091] (2) Performance evaluation metrics:

[0092] Training throughput (Samples per Second): The number of input samples processed by the model per unit time;

[0093] Total communication latency (Total Communication Time);

[0094] Training time per iteration (Iteration Time);

[0095] (3) Evaluation tools:

[0096] The PyTorch Profiler[8] tool was used to collect the communication kernel execution time and GPU running status.

[0097] Monitor GPU utilization and communication bottlenecks using nvidia-smi and NCCL debugging tools.

[0098] (4) Experimental results:

[0099]

[0100] It can be seen from the experimental results that the communication scheduling method proposed by the present invention has significant effects in the training of the mixture of experts model:

[0101] By hiding the All-Reduce operation of the expert layer under the execution of the All-to-All communication, effective overlap of communication tasks is achieved;

[0102] The multi-threaded task distribution mechanism further improves the concurrent execution efficiency;

[0103] Topology awareness and multi-hop path optimization reduce communication congestion and effectively shorten the communication path delay;

[0104] The time consumption of a single round of iteration is shortened by about 5.0%, and the total communication delay is shortened by about 5.37%.

[0105] The above results show that, without introducing additional hardware resources, the present invention can achieve the goals of efficient resource utilization and training acceleration in large-scale model training only through communication scheduling optimization at the software level, especially in communication-intensive parallel training tasks such as MoE. In the future, its scalability and dynamic scheduling adaptation ability can be further verified in a larger-scale training environment (such as multi-machine multi-card, hundreds of GPUs).

[0106] References:

[0107] [1] Fedus W, Zoph B, Shazeer N. Switch transformers: Scaling to trillion-parameter models with simple and efficient sparsity[J]. Journal of Machine Learning Research, 2022, 23(120): 1-39.

[0108] [2] Shazeer N, Mirhoseini A, Maziarz K, et al. Outrageously large neural networks:

[0109] The sparsely-gated mixture-of-experts layer[J].arXiv preprint arXiv:1701.06538,2017.

[0111] [3]Masoudnia S,Ebrahimpour R.Mixture of experts:a literature survey[J].

[0112] Artificial Intelligence Review,2014,42:275-293.

[0113] [4]Lepikhin D,Lee H J,Xu Y,et al.Gshard:Scaling giant models withconditional computation and automatic sharding[J].arXiv preprint arXiv:2006.16668,2020.[5]Nvidia NCCL.Optimized primitives for collective multi-GPUcommunication[EB / OL].(2023).

[0114] [6]Shoeybi M,Patwary M,Puri R,et al.Megatron-lm:Training multi-billion parameter language models using model parallelism[J].arXiv preprintarXiv:1909.08053,2019.

[0116] [7]Rajbhandari S,Li C,Yao Z,et al.Deepspeed-moe:Advancing mixture-of-experts inference and training to power next-generation ai scale[C] / / International conference on machine learning.PMLR,2022:18332-18346.

[0117] [8]Torch Profiler[EB / OL].2024.https: / / pytorch.org / docs / stable / profiler.html。

Claims

1. A communication scheduling method based on a parallel mechanism of hybrid experts, characterized in that It includes three parts: a design load balancing module, a communication resource allocation module, and a parallelization scheduling module. The specific steps are as follows: Step S1: Design the load balancing module, that is, construct an expert model allocation strategy: According to the popularity of the expert model, the computing load, and the resource status of the GPU cluster, dynamically generate the parameter slicing and mapping strategy of the expert model, so as to achieve the efficient coordination and balanced utilization of computing and communication resources; Specifically, an intelligent allocation mechanism is adopted to preferentially map the expert models with frequent computing and large communication requirements to multiple nodes in the form of parameter slicing, so as to provide more computing and communication resources for these expert models; Step S2: Design the communication resource allocation module; that is, design a link resource allocation model for cross-node and intra-node communication, and introduce a multi-hop transmission strategy to achieve the efficient and reasonable allocation of communication link resources; Specifically, when the All-to-All communication spans both inter-machine and intra-machine links at the same time, the data that needs to be transmitted to multiple GPUs within the same node is first centrally sent to a certain GPU on this node, and then this GPU distributes the data to other GPUs within this node; Step S3: Design the parallelization scheduling module, which is used to efficiently coordinate the All-to-All communication between nodes and the All-Reduce communication within nodes, maximize the communication bandwidth utilization rate, and reduce the overall communication delay; Specifically, an asynchronous executor is introduced to make the All-Reduce operation support asynchronous execution, so as to improve the parallelism of communication; At the same time, a multi-thread mechanism is used to block-process the data that needs to perform inter-machine All-to-All and intra-machine All-Reduce communication at the same time, that is, the tensor participating in multiple communication tasks is split into several sub-blocks along a certain dimension, and each sub-block performs communication operations independently, parallelizing the originally serially executed communication process; In this way, while a part of the data is performing inter-machine All-to-All communication, another part of the data is performing intra-machine All-Reduce communication, fully exploiting the concurrent capabilities of the network link.

2. The communication scheduling method based on the parallel mechanism of hybrid experts according to claim 1, wherein The specific process of constructing the expert model allocation strategy described in Step S1 is as follows: First, define which expert models each GPU should be responsible for executing to achieve the load balance of the expert models on the GPUs; According to the given model structure and GPU resources, establish an optimization model with the goal of maximizing the training efficiency; The construction process of the optimization model is as follows: For a single layer containing E expert models, let the number of nodes where the expert models are placed be W, the total number of GPUs be N, and each node be equipped with C homogeneous GPUs. i Indicates the computing power of the GPU where expert i is located, and is expressed in BW m,n represents the communication bandwidth from GPUm to GPUn, and the expert tensor parallelism is T; for the i-th expert model, let x i,n ∈{0,1} indicates whether it is placed on the nth GPU, y i,w ∈{0,1} indicates whether it is placed on the wth host; for simplicity, assume that each input token only selects one expert model for training. According to the output of the gating network, the amount of data that the i-th expert needs to transmit to the j-th expert is D i,j ; The constraints are: (a) Each expert can only be placed within the same physical node, that is, the constraint for the \(i\)-th expert to be placed on the same physical node is (b) To ensure that the computing resources of each node are not overloaded, define the maximum number of GPUs that can be allocated to each node to avoid over-allocation in the case of limited resources, that is, the number of GPUs that can be allocated on the w-th worker node is constrained as: Among them, is used to restrict that the i-th expert is placed within a node and the j-th expert is placed within a node; 1) One expert is located on one GPU: The communication duration from the i-th expert to the j-th expert: The communication duration of the i-th expert: 2) One expert is located on multiple GPUs: The communication duration of the i-th expert: The computing duration of the i-th expert: The optimization problem: minimize T, s.t(a)(b), Among them, represents the calculation duration of the i-th expert; represents the communication duration of the i-th expert.

3. The communication scheduling method based on the parallel mechanism of hybrid experts according to claim 2, wherein In Step S2, the design of the communication link resource allocation model is specifically: Define the communication path from the i-th expert to the j-th expert That is, the communication order is i - k - l - j, and the corresponding bandwidth is Scaling factor Is the proportion of the data transmitted on this path to the total data D i,j Ratio; The limiting conditions are as follows: c) Ensure that the total amount of data transmitted on each communication path is equal to the amount of data that needs to be transmitted between expert models, that is, the total amount of data transmitted in all communication paths from the i-th expert to the j-th expert is equal to D i,j ,Right now d) Ensure that when two expert models are within the same node, their communication paths are only within that node to avoid cross-node communication latency, i.e., the i-th expert and the k-th expert are within the same node; the j-th expert and the l-th expert are within the same node, i.e., y i,w = y k,w , y j,w = y l,w ; e) Ensure that the expert models located in different nodes can communicate synchronously, that is, the i,k-th expert is located in the same node, and the l,j-th expert is located in the same node, that is Among them, used to restrict that the i-th expert is placed within a node and the j-th expert is placed within a node: 1) One expert is located on one GPU: Communication duration from the i-th expert to the j-th expert: Communication duration of the i-th expert: In formula (6), the first term is the transmission time of the data of the \(i\)-th expert himself \(D\) i,j , the second term is the time for transmitting data when at the second node in the path, and the third term is the time for transmitting data when at the third node in the path; \(\beta\) represents the synchronization condition that the \(i\)-th expert and the \(k\)-th expert are in the same node, and the \(l\)-th expert and the \(j\)-th expert are in the same node; when \(\beta = 1\), the constraint of synchronous communication of experts within the above node is satisfied; otherwise \(\beta = 0\); \(\beta_1\) and \(\beta_2\) respectively represent the inter-node synchronization conditions in the second-hop and third-hop stages of the communication path; \(\beta_1 = 1\) indicates that the experts at the second node in the path satisfy the synchronous communication constraint, otherwise \(\beta_1 = 0\); \(\beta_2 = 1\) indicates that the experts at the third node in the path satisfy the synchronous communication constraint, and \(\beta_2 = 0\); 2) One expert is located on multiple GPUs: Communication duration from the i-th expert to the j-th expert: Communication duration of the i-th expert: Computing duration of the i-th expert: Optimization problem: Among them, represents the computing duration of the i-th expert; represents the communication duration of the i-th expert.

4. The communication scheduling method based on the parallel mechanism of hybrid experts according to claim 3, characterized in that In step S3, the design of the parallel scheduling module is to parallelize the inter-machine All-to-All communication and intra-machine All-Reduce communication. The specific method is as follows: (1) Data splitting: Split the tensor that needs to perform All-Reduce and All-to-All operations simultaneously. Specifically, perform All-Reduce operation on half of the data, and then perform All-to-All communication on this part of the data. At the same time, perform All-to-All communication on the other half of the data, and then perform All-Reduce operation on this part of the data. In this way, the overlapping use of resources is achieved during the two communication operations; (2) Parallel scheduling: Through a reasonable scheduling mechanism, ensure that the two communication operations can be performed in parallel, so as to maximize the utilization of link resources and reduce communication latency; Among them, an asynchronous executor is introduced. The asynchronous executor adjusts the primitive execution mode of the underlying communication library to support the asynchronous execution of the All-Reduce key communication operation. At the same time, an efficient thread pool is also introduced to manage the execution of parallel tasks. Each thread in the thread pool is dedicated to handling an independent communication task.

Images