Expert network automatic hybrid distributed algorithm based on cloud side end cross-domain data

Abstract

The invention discloses an expert network automatic hybrid distributed algorithm based on cloud side end cross-domain data, and belongs to the field of large model distributed training. According to the method, an innovative method is introduced in the field of large model distributed training, an OSDP is used for training an MOE model in a distributed mode, distribution of different experts in a GPU cluster is optimized, and higher parallel computing efficiency is ensured. The combined motivation is used for improving the training efficiency under the condition that the model performance is not sacrificed. Through hybrid parallel accelerated training and hardware resource saving, the training cost of a large-scale model is reduced, and training of a super-large-scale deep learning model becomes more economical and feasible. And an automatic hybrid distributed algorithm based on an expert network is introduced, so that cloud, edge and end collaborative cross-domain heterogeneous data fusion is realized, and the complex requirements of the industrial intelligent field in a multi-base and multi-data-source environment can be met.

Description

Technical Field

[0001] The present invention belongs to the field of large-model distributed training, especially in the application of industrial intelligence, and proposes an automatic hybrid distributed algorithm of an expert network based on cross-domain data of cloud-edge-end for cross-domain heterogeneous data fusion of cloud-edge-end. Background Art

[0002] In the field of industrial intelligence, with the rapid development of big data and artificial intelligence, enterprises and industrial sites have accumulated a large amount of heterogeneous and multi-source data, including sensor data, log data, production line data, customer data, etc. The use requirements of a large amount of heterogeneous data and the ever-expanding model scale make it difficult for a single device to meet the training needs of large models. Especially in the field of industrial intelligence, the model may involve a large amount of historical data, real-time monitoring data, etc., which places high demands on computing resources and data processing capabilities. The traditional centralized training method is not only slow, but also easily limited by computing and storage resources. Distributed training has become an effective solution. Through multi-device and multi-node collaborative model training, it can not only improve the training speed, but also speed up the model inference speed, so that the model can better adapt to the real-time and stability requirements of industrial scenarios. In order to obtain better performance when processing multiple data and tasks, many large model training methods have introduced the "expert network" mechanism. The expert network is a modular network structure that divides tasks into different "expert" modules, which focus on different tasks or data types, thereby improving the expressiveness of the model. However, how to effectively use the expert network in distributed training is also a major challenge. Expert networks require efficient communication and coordination mechanisms between modules. In distributed systems, poor coordination may lead to resource waste or performance degradation.

[0003] The combination of MOE and OSDP can effectively cope with the complexity of large-scale deep learning models, ensure efficient training of MOE models on multiple GPUs and nodes, and achieve higher throughput and lower computing costs by optimizing parallel strategies. In the field of distributed deep learning, data parallelism (DP) and zero-redundancy data parallelism (ZDP) are two common parallel training modes, each with its own advantages and disadvantages. The data parallel mode processes data in parallel by copying different parts of the model on multiple devices, thereby accelerating the training process. However, as the size of the model increases, this approach often encounters memory bottlenecks because each device needs to store a complete copy of the model. In contrast, the zero-redundancy data parallel mode significantly reduces memory consumption by retaining only the part of the model that currently needs to be processed on each device (i.e., discarding unnecessary model states). However, this approach comes at the cost of additional communication overhead, because whenever a new part of the model is needed, it needs to be re-collected from other devices. For example Figure 1As shown in the figure, this figure shows the difference in communication operations between ZDP and DP. To process an operator in ZDP mode, three collection operations (two all-gather operations and one reduce-scatter operation) are required, resulting in 3(N-1) communication steps, while DP mode is expected to require 2(N-1) communication steps.

[0004] OSDP balances memory consumption and training efficiency and maximizes the throughput of the entire system. Combining the advantages of DP and ZDP, it intelligently shards models and data and automatically searches for the optimal execution plan to maximize training throughput while meeting device memory constraints. Its core lies in its ability to automatically adjust model sharding, data allocation, and training batch size to find the optimal execution plan. Based on the current model sharding, data allocation, and training batch size, a candidate execution plan is generated and its throughput is predicted. Finally, the plan with the highest estimated system throughput is selected as the final execution plan output. This strategy enables distributed algorithms to maximize training throughput while meeting memory constraints.

[0005] The expansion of model scale will lead to a significant increase in training costs, and the limitation of computing resources has become a bottleneck for large-scale intensive model training. To solve this problem, a deep learning model architecture based on sparse MOE layers is proposed, which splits the large model into multiple small models (experts). Each round of iteration decides to activate a part of the experts for calculation based on the samples, thus saving computing resources. A gate mechanism that is trainable and ensures sparsity is introduced to ensure the optimization of computing power.

[0006] The research motivation of this invention is to provide an effective distributed training method for large-scale deep learning models, improve the scalability and efficiency of distributed training, optimize expert distribution through fine-grained control and intelligent scheduling, maximize the utilization of computing resources, and reduce communication overhead. In order to achieve this goal, this invention proposes an innovative method to use OSDP to distribute the training of expert models. The combination of the two can effectively cope with the complexity of large-scale deep learning models, ensure efficient training of MOE models on multiple GPUs and nodes, and achieve higher throughput by optimizing parallel strategies. Summary of the invention

[0007] The purpose of this invention is to focus on the distributed training and reasoning problems of large models in a collaborative environment of cloud, edge and end. In order to solve the problems of poor scalability and low efficiency of training caused by computing bottlenecks, memory constraints, communication overhead and uneven utilization of hardware resources in large-scale model training, as well as how to reasonably integrate and utilize heterogeneous data, an automatic hybrid distributed algorithm for an expert network based on cloud-edge-end cross-domain data is provided.

[0008] This paper introduces an innovative method in the field of large-scale model distributed training, using OSDP distributed training MOE model to optimize the distribution of different experts in GPU clusters and ensure higher parallel computing efficiency. The motivation for this combination is to improve training efficiency without sacrificing model performance. By hybrid parallel acceleration training and saving hardware resources, the training cost of large-scale models is reduced, making it more economical and feasible to train ultra-large-scale deep learning models.

[0009] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0010] An automatic hybrid distributed algorithm for expert networks based on cloud-edge-end cross-domain data, the algorithm is:

[0011] Step 1: Memory and Communication Quantization Formula

[0012] The optimal execution plan search problem is formulated as follows: given a DL model with n operators, each operator is processed in DP or ZDP mode, the device memory is represented by M_limit, and the OSDP of all operators is and the corresponding batch training size b to search for the optimal parallel mode to minimize the average training time; (M i (p i ,b),T i (p i ,b)) indicates that the i-th operator is in p i Memory and time costs for parallel mode and batch size b; p * represents the optimal execution plan, that is, the optimal parallel mode found, where each operator p i Use a combination of data parallelism (DP) or zero-redundancy data parallelism (ZDP); b. * Represents the optimal execution plan p * The corresponding optimal batch size; the batch size will not exceed the device memory limit M_limit):

[0013]

[0014] To estimate the memory cost, three types of data are considered: model state (including model parameters and optimizer state), intermediate activations, and additional overhead (such as temporary workspace required to operate operators); for the i-th operator, the memory consumed by these three types of data is recorded as three factors respectively Since ZDP divides the model state into multiple parallel processes, the memory consumption of the model state is amortized to 1 / N, where N is the number of parallel processes; therefore, the memory overhead is expressed as

[0015]

[0016] Communication and computational costs are modeled, where α, β, and γ represent network delay, transmission time per byte, and computational coefficient, respectively; the time cost is modeled as

[0017]

[0018] Step 2: Expert network construction

[0019] For a given input, the output of the MOE module is determined by the weighted sum of the outputs of the expert networks, where the weights are given by the outputs of the gating networks; that is, given n expert networks (E0, E1, ..., E n-1 ), then the output of the expert layer is G(x) i represents the n-dimensional output of the gating network of the i-th expert, E i (x) is the output of the i-th expert network;

[0020] Step 3: Estimate memory and time costs:

[0021] (1) M(p,b): Estimates memory requirements based on batch size b and current expert selection;

[0022] (2) T(p,b): estimates the total time to execute plan p, including the computation time of forward propagation and backward propagation, and the communication overhead between multiple experts;

[0023] Traverse the execution plan through depth-first search: If the memory limit satisfies M(p,b)≤M limit And T(p,b) <T * (b)

[0024] Update the current optimal plan: T * (b)←T(p,b),p * (b)←p;

[0025] If no feasible plan p is found for the current batch size b * (b) = None, then stop searching: all plans under the current batch size exceed the memory limit; if a feasible plan is found, add it to the candidate plan set P: P←P∪{(T * (b),p * (b),b)};

[0026] Choose the time-optimal schedule and batch size: p * ,b * ←argmin p,b {T(b)|(T(b),p,b)∈P}

[0027] Returns the optimal execution plan p * and the corresponding batch size b * .

[0028] Furthermore, in step 2, G(x) is realized i There are many methods to implement a simple and efficient method by taking softmax on the Top-Klogits of the linear layer; the method of the present invention to implement G(x) is as follows:

[0029] G(x):=Softmax(TopK(x*W g )), where x is the input feature vector, usually a representation from the previous network layer or input data; W g is the weight matrix of the gating network. The function of this matrix is ​​to generate expert activation scores by linearly transforming the input feature vector x; K is the parameter selected by TopK, which means selecting the K experts with the highest scores.

[0030] The beneficial effects of the present invention relative to the prior art are as follows: the present invention proposes a combination of MOE+OSDP, which improves the training efficiency of deep learning models on large-scale GPU clusters through automated parallel optimization and efficient scheduling of expert distribution. The first problem solved by the present invention is the training efficiency problem of large-scale models. Through MOE (Mixture of Experts), the model will only activate a few experts to participate in the calculation, reducing the number of activated parameters, thereby improving the training efficiency. At the same time, OSDP intelligently selects and configures the mixed parallel mode of the model through DFS without exceeding the memory limit, further accelerating the training process. The second problem solved is the communication overhead in the expert model. The experts of the MOE model may be distributed on different GPUs, and frequent communication across GPUs will cause huge communication overhead, especially when training in large-scale clusters, the communication bandwidth becomes a bottleneck. OSDP can reasonably distribute MOE experts among multiple GPUs and reduce the communication load between GPUs through automated parallel scheduling. It optimizes the distribution of experts and intelligently arranges tensor parallelism and pipeline parallelism, thereby reducing communication overhead and improving the efficiency of data transmission. The third problem solved is memory bottleneck and insufficient resources. In large-scale model training, GPU memory resources are limited. Especially when the model scale is very large, a single GPU cannot accommodate the parameters of the entire model, resulting in OOM (Out of Memory) errors. The MOE model effectively reduces memory usage by activating only a small number of experts in each forward calculation process. OSDP can automatically select the best parallel strategy based on the GPU memory constraints, avoid memory overload, and ensure the smooth progress of the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the communication operation diagram of ZDP and DP;

[0032] Figure 2 It is the expert model diagram;

[0033] Figure 3 This is the OSDP architecture diagram;

[0034] Figure 4 This is the result diagram of Example 1;

[0035] Figure 5 This is the result diagram of Example 2. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific implementation methods. It should be understood that the specific implementation methods described here are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0037] The following is an explanation of the professional terms used in this case:

[0038] Large Model Distributed Training: With the continuous development of deep learning technology, the scale of models has gradually increased, and the demand for computing resources has also increased. The basic principle of distributed training is to distribute the data and computing tasks of large models to multiple computing nodes, with each node responsible for processing a part of the data and computing tasks. Through the communication network, gradients and parameters can be shared between nodes to achieve synchronous updates of the model. This method can make full use of computing resources, accelerate the training process, and improve the performance of the model.

[0039] Mixture of Experts: Mixture of Experts is an innovative deep learning architecture. Figure 2 It combines multiple expert models (usually neural networks) with gating networks to achieve higher computational efficiency and better performance by decomposing complex tasks into multiple subtasks and assigning them to different expert models for processing.

[0040] Automatic hybrid distributed parallelism: It combines various distributed parallel methods and implements automated or semi-automated parallel processes to optimize the training efficiency and performance of deep learning models. After the user is given a model and the machine resources used, it can automatically help the user choose a better or optimal parallel strategy for efficient execution. It combines the advantages of multiple parallel methods and automatically or semi-automatically selects the appropriate parallel strategy according to specific scenarios and needs.

[0041] Optimal Sharded Data Parallel for Distributed Deep Learning: OSDP is a parallel framework designed specifically for large-scale distributed deep learning. Figure 3 As shown in Figure 1, it aims to improve the training efficiency of deep learning models in multi-node and multi-GPU environments. It optimizes the model training process through flexible and automated hybrid parallel strategies, and can make intelligent scheduling and selection between different parallel dimensions (such as data parallelism, tensor parallelism, and pipeline parallelism).

[0042] Specific implementation method 1: This implementation method describes an automatic hybrid distributed algorithm based on an expert network, and the algorithm is:

[0043] Step 1: Memory and Communication Quantization Formula

[0044] For formalization, the optimal execution plan search problem is formulated as follows: Given a DL model with n operators, where each operator is processed in DP or ZDP mode, the device memory is represented by M_limit, and the OSDP is the number of all operators. and the corresponding batch training size b to search for the optimal parallel mode to minimize the average training time;

[0045] (M i (p i ,b),T i (p i ,b)) indicates that the i-th operator is in p i Memory and time costs for parallel mode and batch size b; p * represents the optimal execution plan, that is, the optimal parallel mode found, where each operator p i Use a combination of data parallelism (DP) or zero-redundancy data parallelism (ZDP); b. * Represents the optimal execution plan p * The corresponding optimal batch size; the batch size will not exceed the device memory limit M_limit):

[0046]

[0047] This paper introduces how OSDP estimates memory and time costs respectively. To estimate memory cost, three types of data are considered: model state (including model parameters and optimizer state), intermediate activations, and additional overhead (such as temporary workspace required to operate operators); for simplicity, for the i-th operator, the memory consumed by these three types of data is recorded as three factors Since ZDP divides the model state into multiple parallel processes, the memory consumption of the model state can be amortized to 1 / N, where N is the number of parallel processes; therefore, the memory overhead is expressed as

[0048]

[0049] The time cost consists of communication cost and computational cost, where the communication cost is related to the number of model parameters, while the computational cost is related to the training batch size; therefore, the communication and computational costs are modeled, α, β, γ represent network delay, transmission time per byte, and computational coefficient, respectively; to complete an all-gather or reduce-scatter operation, N-1 communication steps are required, and the communication volume of each step is S i / N, where N is the number of parallel processes, S i is the size of the model parameters of the i-th operator; if an operator is processed in ZDP mode, three collective operations (two all-gather operations and one reduce-scatter operation) are required, resulting in 3(N-1) communication steps, while DP mode is expected to require 2(N-1) communication steps; as for the computational cost, it is proportional to the training batch size and the computation coefficient; putting them together, the time cost is modeled as

[0050]

[0051] Step 2: Expert network construction

[0052] Mixtral is a sparse mixture of experts network. It is a decoder-only model in which the feedforward block picks from 8 different sets of parameters; at each layer, for each token, a router network selects two of these groups ("experts") to process the token and sums their outputs; this technique increases the number of parameters of the model while controlling cost and latency, because the model only uses a small fraction of the total set of parameters for each token. For a given input, the output of the MOE module is determined by the weighted sum of the outputs of the expert networks, where the weights are given by the outputs of the gating network; that is, given n expert networks (E0, E1, ..., E n-1 ), then the output of the expert layer is G(x) i represents the n-dimensional output of the gating network of the i-th expert, E i (x) is the output of the i-th expert network; realize G(x) i There are many methods. A simple and efficient method is to take softmax on the Top-Klogits of the linear layer. The method of the present invention to implement G(x) is as follows: G(x): = Softmax(TopK(x*W g)), where x is the input feature vector, usually a representation from the previous network layer or input data; W g is the weight matrix of the gating network. The function of this matrix is ​​to generate expert activation scores by linearly transforming the input feature vector x; K is the parameter selected by TopK, which means selecting the K experts with the highest scores.

[0053] Step 3: OSDP System Overview

[0054] The workflow of OSDP consists of three main modules: Memory Time Estimator, Plan Generator, and Scheduler. These modules work together to adaptively and automatically maximize the overall system throughput. Below we will introduce the functions of these modules in depth.

[0055] Memory Time Estimator: Responsible for evaluating the memory and time costs of a given execution plan and batch size (combined with the model description and device information provided by the user). It outputs the estimated memory cost and time cost for a specific execution plan and batch size. This information is the basis for subsequent optimization steps.

[0056] Plan Generator: It uses the cost estimate provided by the memory time estimator to find the optimal execution plan through depth-first search (DFS); DFS allows it to systematically traverse all possible execution plan combinations while making decisions based on memory and time costs; its optimization strategy is to first ensure that the generated execution plan does not exceed the memory limit of the device in terms of memory cost; then, within the memory limit, it selects the execution plan with the smallest overall time cost; finally, it introduces two pruning strategies to speed up the search process: if the current memory usage exceeds the limit or the current time cost exceeds the known best plan, it stops searching the branch further; it outputs the optimal execution plan and its corresponding batch size, as well as the system throughput estimate under this plan;

[0057] Scheduler: The scheduler is responsible for iteratively collecting candidate execution plans and throughput estimates from the plan generator, and gradually increasing the training batch size until the memory limit is reached; its goal is to find an execution plan that maximizes the system throughput; the scheduler tries to make full use of device memory every time the batch size is adjusted, which helps to achieve higher throughput with smaller batch sizes; among all candidate plans, the scheduler selects the plan with the highest estimated system throughput as the final execution plan; the scheduler outputs the overall optimal execution plan, which can achieve the highest system throughput under the given memory limit;

[0058] This architecture solves the problem of optimizing the deep learning model training process under given memory constraints through the collaborative work of the memory time estimator, plan generator, and scheduler. Through depth-first search and intelligent pruning strategies, the plan generator can quickly find the optimal execution plan, while the scheduler ensures that the plan can maximize system throughput.

[0059] Estimated memory and time costs:

[0060] (1) M(p,b): Estimates memory requirements based on batch size b and current expert selection;

[0061] (2) T(p,b): estimates the total time to execute plan p, including the computation time of forward propagation and backward propagation, and the communication overhead between multiple experts;

[0062] Traverse the execution plan through depth-first search: If the memory limit satisfies M(p,b)≤M limit And T(p,b) <T * (b): Update the current optimal plan: T * (b)←T(p,b),p * (b)←p;

[0063] If no feasible plan p is found for the current batch size b * (b) = None, then stop searching: all plans under the current batch size exceed the memory limit; if a feasible plan is found, add it to the candidate plan set P: P←P∪{(T * (b),p * (b),b)};

[0064] Choose the time-optimal schedule and batch size: p * ,b * ←arg min p,b {T(b)|(T(b),p,b)∈P}

[0065] Returns the optimal execution plan p * and the corresponding batch size b * .

[0066] Step 4: MOE+OSDP

[0067] Combining MOE and OSDP, we can build an efficient distributed deep learning framework that can provide higher computing efficiency and model performance when processing large-scale data and complex tasks. This framework consists of three main parts: gating network, expert network and automated sharding distributed scheduling architecture.

[0068] The core task of the gating network is to select the appropriate expert network to handle specific tasks based on the characteristics of the input data. It is the intelligent scheduling center of the entire system, responsible for dynamically assigning tasks to the most suitable expert model based on the characteristics of the input data. The gating network receives input data from the master node and generates a probability vector based on the characteristics of this data. This vector represents the activation probability (i.e., selection weight) of different expert networks. For example, if there are 10 expert networks, the output of the gating network may be a vector of length 10, each value representing the probability of the expert being activated. The output of the gating network will serve as the basis for task allocation to determine which expert networks process which subsets of input data. Through this mechanism, the framework can dynamically adjust the calculation path under different tasks and data characteristics, avoiding all expert networks from processing each task at the same time, thereby improving efficiency.

[0069] Expert networks are responsible for specific computational tasks. Each expert network is an independent neural network that is trained specifically on a specific subset of data or task. During the training phase, each expert network learns to process a specific type of data or solve a specific subtask to improve the efficiency and accuracy of the overall system. Each expert network is trained on an independent subset of data, focusing on a specific type of input pattern or task. This reduces the burden on a single network, improves the generalization ability of the model, and increases the training speed. After training, the expert networks are deployed on multiple worker nodes. Each worker node is responsible for one or more expert networks and performs the corresponding computational tasks in the inference phase.

[0070] The automated sharding distributed scheduling architecture is used to efficiently distribute and synchronize the computing tasks of the expert network, ensuring that each node reasonably distributes the computing load and dynamically performs data transmission and gradient synchronization during the computing process. It divides the computing tasks of the expert network into multiple small subtasks and assigns them to multiple nodes or GPUs for execution. This sharding technology allows each node or GPU to only process its assigned task subset, reducing the computing pressure and video memory usage of a single device.

[0071] The following is a detailed training procedure. The input of the algorithm is the model description (MD) and the device information (DI). The output is the optimal execution plan p*, corresponding to the batch size b.

[0072] 1. Initialize the candidate execution plan set: P←{}

[0073] 2. Select active experts for each forward propagation:

[0074] (1) The gating network generates a gate for each expert e based on the characteristics of the input data x. i Calculate the activation score g i (x).

[0075] (2) Select the top-K experts with the highest scores for calculation

[0076] 3. Traverse different batch sizes b∈{1,2,3,…}

[0077] Initialize the optimal time and plan for the current batch size:

[0078] T * (b)←INF

[0079] p * (b)←None

[0080] 4. Traverse all possible execution plans p∈{DP,ZDP} n :

[0081] Each plan may include a combination of data parallelism (DP) and zero-data parallelism (ZDP).

[0082] 5. Estimate memory and time costs:

[0083] (1) M(p,b): Estimates the memory requirements based on the batch size b and the current expert selection.

[0084] (2) T(p,b): Estimates the total time to execute plan p, including the computation time of forward propagation and backward propagation, and the communication overhead between multiple experts.

[0085] 6. Traverse the execution plan through depth-first search:

[0086] If the memory limit satisfies M(p,b)≤M limit And T(p,b) <T * (b)

[0087] Update the current optimal plan:

[0088] T * (b)←T(p,b)

[0089] p * (b)←p

[0090] 7. If no feasible plan p is found for the current batch size b * (b) = None, then stop searching: all plans exceed the memory limit under the current batch size.

[0091] 8. If a feasible plan is found, add it to the candidate plan set P: P←P∪{(T * (b),p * (b),b)}

[0092] 9. Choose the optimal time plan and batch size: p * ,b* ←arg min p,b {T(b)|(T(b),p,b)∈P}

[0093] 10. Return the optimal execution plan p * and the corresponding batch size b * .

[0094] The framework sets up a master node to be responsible for the calculation of the gating network and the scheduling and coordination of tasks among multiple worker nodes. Each worker node runs one or more expert networks and performs specific computing tasks. Each expert network is trained separately on a specific data subset or task. When new data arrives, the master node first performs necessary preprocessing steps, such as normalization and feature extraction, to ensure that the data can be efficiently processed by the gating network and the expert network. The preprocessed data is input to the gating network, which generates a probability vector based on the characteristics of the input data, which represents the activation probability of each expert network. Specifically, the gating network learns how to select the most appropriate expert based on different input data, and the expert model with a higher probability will be preferentially selected to process the task. The master node distributes the input data to the expert networks on different worker nodes in proportion based on the output of the gating network. This task scheduling process is dynamic, ensuring that the input data is processed by the most appropriate expert network to improve computing efficiency and model performance. After receiving the task, each expert network in the worker node performs calculations on its independent data subset or task. Through the automated sharding distributed architecture, the calculation of the task is distributed to different nodes and GPUs for parallel execution. During the training process, the expert network can quickly adapt and optimize its performance on the task by training separately on its specific subset of data. This makes the expert network more robust to different types of data. After completing the expert network calculation, each working node returns the result to the master node. At this time, the automated sharded distributed scheduling architecture is responsible for efficiently synchronizing data between nodes, ensuring that tasks are completed on time and performing necessary redundant operations to prevent the failure of a node from affecting the overall result. The master node performs weighted fusion of the calculation results of each expert network based on the weights output by the gating network, and finally obtains the prediction result of the overall model. This ensures that the results of different experts generate the optimal output after comprehensive evaluation.

[0095] Example 1

[0096] Based on the specific implementation method one, this embodiment is carried out on a laboratory server equipped with 8 NVIDIA GeForce RTX3090 GPUs, and the upper limit of the device memory is set to 8G. Compare MOE+OSDP with pure parallel and hybrid parallel strategies. For pure parallel strategies, PyTorch DDP is selected as the representative of DP. FairScale is selected as the representative of FSDP / ZeRO. The hybrid parallel strategy selects MOE+DP. In order to achieve a fair comparison, the present invention optimizes the hybrid parallelization parallel strategy combination and reports the one with the best performance. The present invention uses GPT as the experimental model library, which is divided into gpt-0.3b, gpt-1.5b, and gpt-2.7b.

[0097] like Figure 4 As shown in the figure, MOE+OSDP has higher throughput than other strategies on all model types, and traditional data parallelism (DP) performs the worst because it needs to synchronize parameters across all GPUs. FSDP provides good throughput, but its performance degrades on complex models due to the synchronization overhead of full sharding. MOE+DP is slightly better than FSDP, but the throughput is not fully improved due to the lack of further optimization of expert allocation and load balancing.

[0098] Example 2

[0099] Based on the first specific implementation, this example is carried out on a laboratory server equipped with 8 NVIDIA GeForce RTX3090 GPUs, and the upper limit of the device memory is set to 8G. This example performs distributed training under different numbers of GPUs. The model uses GPT-1.5b, and the distributed training method selects MOE+OSDP and OSDP.

[0100] like Figure 5 As shown in the figure, MOE+OSDP shows an accelerating growth trend: the throughput increases with the number of GPUs. This shows that this method performs well under high GPU configurations and can effectively utilize resources. The scalability of OSDP gradually slows down, which shows that although OSDP can be extended to multiple GPUs, its efficiency is not as good as MOE+OSDP after the number of GPUs increases. MOE+OSDP has better scalability on multi-GPU clusters and can still run efficiently even on large-scale clusters.

[0101] MOE+OSDP can achieve better load balancing. By dynamically allocating expert networks, it reduces the memory bottleneck and computing pressure of a single GPU. MOE+OSDP provides higher training throughput, especially on large models, which is significantly improved compared to traditional strategies such as FSDP and DP. When the number of GPUs increases, MOE+OSDP can effectively expand throughput and make full use of multi-GPU resources. Through the dynamic allocation of expert networks, the computing resources of each GPU are balanced, avoiding the problem of overload or idleness of some GPUs. The advantage of MOE+OSDP lies in its automatic search for hybrid parallel strategies, while optimizing the distribution and communication efficiency of expert networks.

Claims

1. An automatic hybrid distributed algorithm for expert networks based on cloud-edge-end cross-domain data, characterized by: The algorithm is: Step 1: Memory and Communication Quantization Formula The optimal execution plan search problem is formulated as follows: given a DL model with n operators, each operator is processed in DP or ZDP mode, the device memory is represented by M_limit, and the OSDP of all operators is and the corresponding batch training size b to search for the optimal parallel mode to minimize the average training time; (M i (p i ,b),T i (p i ,b)) indicates that the i-th operator is in p i Memory and time costs for parallel mode and batch size b; p * represents the optimal execution plan, that is, the optimal parallel mode found, where each operator p i Use data parallelism or a combination of zero-redundancy data parallelism; b. * Represents the optimal execution plan p * The corresponding optimal batch size; the batch size will not exceed the device memory limit M_limit): To estimate the memory cost, three types of data are considered: model state (including model parameters and optimizer state), intermediate activations, and additional overhead (such as temporary workspace required to operate operators); for the i-th operator, the memory consumed by these three types of data is recorded as three factors respectively Since ZDP divides the model state into multiple parallel processes, the memory consumption of the model state is amortized to 1 / N, where N is the number of parallel processes; therefore, the memory overhead is expressed as Communication and computational costs are modeled, where α, β, and γ represent network delay, transmission time per byte, and computational coefficient, respectively; the time cost is modeled as Step 2: Expert network construction For a given input, the output of the MOE module is determined by the weighted sum of the outputs of the expert networks, where the weights are given by the outputs of the gating networks; that is, given n expert networks (E0, E1, ..., E n-1 ), then the output of the expert layer is G(x) i represents the n-dimensional output of the gating network of the i-th expert, E i (x) is the output of the i-th expert network; Step 3: Estimate memory and time costs: (1) M(p,b): Estimates memory requirements based on batch size b and current expert selection; (2) T(p,b): estimates the total time to execute plan p, including the computation time of forward propagation and backward propagation, and the communication overhead between multiple experts; Traverse the execution plan through depth-first search: If the memory limit satisfies M(p,b)≤M limit And T(p,b) <T * (b): Update the current optimal plan: T * (b)←T(p,b),p * (b)←p; If no feasible plan p is found for the current batch size b * (b) = None, then stop searching: all plans under the current batch size exceed the memory limit; if a feasible plan is found, add it to the candidate plan set P: P←P∪{(T * (b),p * (b),b)}; Choose the time-optimal schedule and batch size: p * ,b * ←argmin p,b {T(b)|(T(b),p,b)∈P} Returns the optimal execution plan p * and the corresponding batch size b * .

2. According to claim 1, an expert network automatic hybrid distributed algorithm based on cloud-edge-end cross-domain data is characterized by: In step 2, the method to implement G(x) is as follows: G(x): = Softmax(TopK(x*W g )), where x is the input feature vector, usually a representation from the previous network layer or input data; W g is the weight matrix of the gating network. The function of this matrix is ​​to generate expert activation scores by linearly transforming the input feature vector x; K is the parameter selected by TopK, which means selecting the K experts with the highest scores.

Images