Hybrid expert model training optimization method based on expert calculation load balancing scheduling

Abstract

The invention discloses a hybrid expert model training optimization method based on expert calculation load balancing scheduling, and the method comprises the steps: firstly predicting current iteration global expert load data, and obtaining sampling data; and secondly, designing a multi-dimensional performance perception expert placement evaluation method, and realizing quantitative comparison and decision in a stage of searching an expert placement scheme by converting multi-dimensional communication and calculation into a unified time delay index. And then designing an expert placement strategy by adopting an expert grouping search algorithm of dynamic load feature perception and a matching algorithm of expert grouping and resource perception based on the sampling data and an expert placement evaluation method. Finally, on the basis of an expert placement strategy, an asynchronous communication strategy is adopted to implement expert and optimizer parameter scheduling, and training optimization is completed. According to the method, by improving the computing load balance and asynchronous communication concealment between the devices, synchronous waiting is reduced, and the training efficiency and the overall performance of the hybrid expert model are effectively enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large model distributed computing, and particularly relates to a hybrid expert model training optimization method based on expert computing load balancing scheduling. BACKGROUND

[0002] In recent years, with the continuous growth of data volume and computing power, large models based on Transformer have made significant breakthroughs in natural language processing, computer vision and other fields. These models rely on large-scale parameters, extensive data sets and powerful computing resources, and exhibit superior performance on a variety of complex tasks. However, traditional dense models have high computing requirements and poor scalability, while MoE-based models are a sparse architecture that achieves comparable performance to dense models with less computing power and stronger scalability, and are therefore widely used in large model design. The core of the MoE model consists of a group of "expert networks" and a "gate network". Each MoE layer contains multiple independent expert networks, which focus on different tasks or input features. The gate network is responsible for selecting the most suitable expert according to the input features, thereby controlling which experts are activated in model inference or training. Such a design allows the model to achieve efficient sparse computing by activating part of the experts, thereby expanding the parameter size and capacity of the model without significantly increasing the computing cost.

[0003] At the same time, with the increase in model parameters, the demand for device memory during model training increases proportionally. To solve this problem, the EP (Expert Parallelism) technique is usually used in the training of MoE models, which distributes the parameters of the experts in the MoE layer to different computing devices, and each device is responsible for storing and calculating a part of the expert weights. When the model is trained, based on the decision of the gate network, only the selected experts will be activated, and the expert input is transmitted to different devices through Allgather or AlltoAll communication. This design avoids storing all expert parameters on a single device, and the calculation of activated experts can also be parallelized across computing devices, solving the memory bottleneck problem while improving overall computing efficiency.

[0004] However, in large-scale MoE models, the input of the MoE layer is very uneven for the selection preference of the experts, and the more popular experts will get more input data than other experts, which will cause some experts to be overloaded while others are idle. This load imbalance not only reduces the utilization of hardware resources, but also causes synchronization waiting of MoE part communication and calculation between devices, thereby affecting the overall training efficiency. More complex is that this imbalance has dynamic change characteristics, and using a pre-set distributed parallel training strategy cannot always get better system resource utilization. Therefore, how to efficiently distribute the MoE model for training without affecting the model quality to achieve maximum efficiency is still a problem to be solved. Although the MoE model uses the Allgather communication mode for training to obtain all input data on each device, it solves the communication load imbalance problem of the AlltoAll mode in the expert parallel scene, but there is still a problem of expert computing load imbalance caused by the preference of input to expert selection. SUMMARY

[0005] In view of the problem of training efficiency reduction caused by expert computing load imbalance in the MoE model Allgather communication scene, the present application provides a mixed expert model training optimization method based on expert computing load balancing scheduling. The method uses an expert grouping search algorithm based on dynamic load characteristic perception and a matching algorithm based on expert grouping and resource perception to comprehensively search for expert placement strategies and schedule experts. The communication and calculation overheads generated in the expert scheduling process are analyzed in depth, and the performance benefits brought by scheduling are comprehensively evaluated to ensure high utilization of system resources during model training. Specifically, the method includes the following steps:

[0006] Step one: predict the global expert load data of the current iteration based on the expert load prediction mechanism to obtain sampling data. Since the expert load is dynamically determined by the gating network according to the allocation of the input, and under the data parallel strategy, each device can only observe the local gating output, therefore, after the gating network calculation in the mixed expert model, the expert selection results of each device are aggregated by using a set communication operation (such as Allgather), and the historical global expert load data and expert fluctuation characteristic data are sampled. The expert fluctuation characteristic data is the standard deviation of the historical global expert load data.

[0007] Based on such sampling data, the present application designs an expert load prediction evaluation model, which uses historical global expert load data to recursively predict the global expert load of the current iteration according to the weight index a At the same time, in order to enhance the adaptability of the prediction evaluation model to the change of the expert load, the adaptive updating mechanism of the weight index is further adopted to calculate the predicted value after each iteration of the model the error between the real value L (t) between the real value L

[0008] The historical global expert load data, expert fluctuation feature data and predicted global expert load constitute the sampling data.

[0009] The prediction evaluation model predicts accurate and reliable global expert load data, providing basic data support for subsequent scheduling and placement optimization.

[0010] Step two: based on hardware device information and related theoretical analysis, the application designs a multi-dimensional performance-aware expert placement evaluation method, which converts multi-dimensional communication and calculation into a unified latency index, realizes quantitative comparison and decision-making in the search expert placement scheme stage.

[0011] In terms of benefits, the application uses a nonlinear function to model the impact of load on computing latency, compares the maximum expert computing load change of each GPU-accelerator (graphics processing unit GPU) before and after expert scheduling, and evaluates the scheduling benefits. The communication overhead is mainly derived from the scheduling of expert parameters and optimizer parameters, and the total scheduling benefit is obtained by subtracting the communication overhead from the scheduling benefit. When the total scheduling benefit exceeds the preset threshold, it is determined that the current scheduling needs to be performed.

[0012] Step three: based on the sampling data of step one and the expert placement evaluation method of step two, the application uses an expert grouping search algorithm based on dynamic load feature perception and a matching algorithm based on expert grouping and resource perception to design an expert placement strategy to realize cross-device load balancing and reduce communication overhead. First, based on the sampling data of step one, the expert grouping search algorithm based on dynamic load feature perception is used to group the experts, and the expert fusion grouping that realizes load balancing is realized. Second, the matching algorithm based on expert grouping and resource perception is used to realize the optimal matching of expert grouping and GPU-accelerator, so as to realize the minimum total scheduling cost of expert placement scheme and finally obtain the scheme with the maximum overall benefit.

[0013] The specific method for realizing expert group load balancing by the expert group search algorithm based on dynamic load feature sensing is that, based on the sampling data of step one, the experts are placed into the group with the minimum comprehensive cost one by one while keeping the number of final experts in different groups consistent, the cost considering the current total load and fluctuation accumulation of the group, and finally obtaining an expert combination scheme with smaller load difference between groups, and calculating the benefit according to the expert placement evaluation method of step two.

[0014] Secondly, the specific method for realizing optimal matching between the expert group and the GPU-accelerator by the matching algorithm based on expert group and resource sensing is that: first, the scheduling cost matrix is constructed according to the overhead evaluation method and the expert group, and the cost array of the expert group and the GPU-accelerator is initialized to represent the acceptable cost concession in the allocation process. Then in each iteration, for each expert group, the algorithm selects the device with the lowest cost in the current iteration based on the cost matrix to perform allocation attempt, and after the allocation is completed, the system updates the cost array and the cost matrix according to the allocation result of this round. The process is iterated until the mapping relationship between the expert group and the GPU-accelerator tends to be stable or reaches the preset iteration upper limit, and finally realizes the mapping scheme with the minimum scheduling overhead between the expert group and the GPU-accelerator.

[0015] Step four: based on the expert placement strategy of step three, the asynchronous communication strategy is used to implement the scheduling of experts and optimizer parameters to realize the masking of communication scheduling, reduce the communication overhead and complete the training optimization. By comparing the new and old expert placement strategies, the parameters to be scheduled are extracted, and the AlltoAll communication operation is used for inter-device scheduling. In order to reduce the influence of communication on training performance, the asynchronous communication strategy is adopted in the scheduling process, and the scheduling process of the expert and optimizer parameters is embedded in the Attention layer and the FFN layer respectively. Since these calculation stages have no dependence on communication operations, they can be executed in parallel, thereby effectively masking the communication delay, improving the device utilization and overall training efficiency.

[0016] The present application has the following advantages:

[0017] The present application proposes a hybrid expert model training optimization method based on expert computing load balancing scheduling to solve the problem of training efficiency decline caused by unbalanced expert computing load in the MoE model Allgather communication scenario. The method samples the MoE layer expert load feature data in the training process, dynamically schedules experts between devices using the asynchronous communication strategy, improves the computing load balancing between devices and the asynchronous communication masking, reduces the synchronous waiting time, and effectively enhances the training efficiency and overall system performance. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The present application has the following advantages:

[0017] The present application proposes a hybrid expert model training optimization method based on expert computing load balancing scheduling to solve the problem of training efficiency decline caused by unbalanced expert computing load in the MoE model Allgather communication scenario. The method samples the MoE layer expert load feature data in the training process, dynamically schedules experts between devices using the asynchronous communication strategy, improves the computing load balancing between devices and the asynchronous communication masking, reduces the synchronous waiting time, and effectively enhances the training efficiency and overall system performance. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 The present application has the following advantages: DETAILED DESCRIPTION

[0019] A hybrid expert model training optimization method based on expert computing load balancing scheduling, as shown in the flow in the specification, specifically comprising the following steps: Figure 1

[0020] Step one: predict the global expert load data based on the sampling expert load prediction mechanism, in order to realize the load balancing between devices, it is necessary to reasonably estimate the load distribution of each expert in each iteration. Since the expert load is dynamically determined by the gating network according to the input allocation result, and under the data parallel strategy, each device can only observe the local gating output, therefore, after the gating network calculation, the collection communication operation (such as Allgather) is used to summarize the expert selection results of each device, and the historical global expert load data and expert load fluctuation characteristic data are sampled.

[0021] Based on such sampling data, the present application designs an expert load prediction evaluation model, which uses historical global load data to recursively predict the global expert load of the current iteration according to the weight index

[0022]

[0023] Where L (t-1) is the true expert load sampled in the last iteration, is the prediction result of the last round, and a is the weight index, which is used to control the calculation weight of historical load and past predicted load.

[0024] At the same time, in order to enhance the adaptability of the prediction evaluation model to the change of expert load, the adaptive updating mechanism of weight index is further adopted. Specifically, the model will calculate the error between the predicted value and the true value after each iteration, and dynamically adjust a according to the error size, so that the prediction evaluation model responds faster when the load fluctuates sharply, thereby improving the responsiveness and robustness of the prediction. The adjustment formula is as follows:

[0025]

[0026] Where is the prediction result of the current iteration, L (t) is the true expert load sampled in the current iteration, and β is the adjustment weight, and ε is a small stable term to prevent the denominator from being 0. The historical global expert load data, expert fluctuation characteristic data and predicted global expert load constitute the sampling data.

[0027] ​In combination with the above process, the prediction evaluation model obtains accurate and reliable global expert load data, providing basic prior data support for subsequent scheduling and placement optimization, so that the scheduling process can be placed in longer calculations to mask the scheduling overhead. In combination with the characteristics of the model, the scheduling process is placed in the Attention layer and the FFN layer calculation of MoE.

[0028] Step two: Based on the hardware device information and related theoretical analysis, the system designs a multi-dimensional performance-aware expert placement evaluation method, which converts multi-dimensional communication and calculation into a unified time index to accurately analyze the impact of different placement schemes on overall training performance.

[0029] The benefit of expert scheduling comes from the reduction in maximum computational load between devices caused by the difference in expert placement before and after optimization, thereby reducing the overall computation time. Since the actual computation performance of GPU-accelerator increases nonlinearly with the increase of load, a nonlinear function f(x) = x log(1 + theta x) is used to simulate the impact of load change on computation time, where theta is the adjustment factor. Let Load i be the load of the ith device, gamma be the computation amount of a single data for expert computation, and F be the effective computing power. Therefore, the scheduling benefit can be expressed as:

[0030]

[0031] Load

[0032] The scheduling overhead comes from the scheduling of expert parameters and optimizer states caused by the change in expert placement, and the overhead of the two needs to be calculated by comparing the placement schemes before and after optimization.

[0033] 1. Expert parameter scheduling overhead: Let the number of experts to be scheduled on the ith device be N i , the size of the expert parameter be Shape, the data type size be Size, the communication bandwidth be H, and the communication startup delay be tau. Then the expert parameter scheduling communication is represented as:

[0034]

[0035] Considering that the expert scheduling communication overlaps with the Attention calculation process, the expert parameter scheduling overhead is represented as:

[0036]

[0037] Where seq is the sequence length of the Attention layer, kappa is the unit time consumption coefficient, and epsilon is a stabilizing term to prevent the denominator from being zero.

[0038] 2. Optimizer parameter scheduling overhead: Similar to the expert scheduling, the optimizer part needs to schedule the mean and variance of each expert parameter gradient to update the expert parameters correctly, so the optimizer parameter scheduling communication Opti comm is equal to 2*Expert comm . The optimizer scheduling communication overlaps with the MoE FFN layer computation, and the optimizer scheduling overhead is denoted as:

[0039]

[0040] Based on the above evaluation of multiple communications and computation time, the final search for C total is the maximum and greater than the set threshold, otherwise the expert remains unchanged.

[0041] C total = T profit - Expert overhead - Opti overhead (7)

[0042] Step three: expert grouping search based on dynamic load feature perception and matching algorithm search for expert placement scheme based on expert grouping and resource perception. In this stage, the application first searches for a suitable expert grouping strategy based on expert load and load fluctuation characteristics, and comprehensively realizes load balancing expert fusion grouping. Algorithm 1 shows the detailed process of expert grouping. First, initialize several empty expert groups, and record the current cumulative total cost of each group. In the allocation process, the current expert will be allocated to the group with the minimum total cost, which considers the current total load and fluctuation accumulation of the group, and the calculation formula is P = e + μ·σ, where e and σ are the predicted global expert load and expert fluctuation characteristic total value in the group, and μ is the fitting parameter. Update the cost state of the target group synchronously after each allocation until all experts are grouped. Through this grouping strategy, the maximum load difference between groups can be effectively reduced and the stability of the group can be maintained, and a load balanced expert group is constructed. After expert grouping, the scheduling benefit T profit is calculated according to formula (3).

[0043]

[0044] After obtaining the load-balanced expert groups according to the above algorithm, each expert group needs to be mapped to a specific GPU-accelerator for execution. To reduce unnecessary expert scheduling overhead, this invention adopts a matching algorithm based on expert groups and resource awareness to search for a scheme that minimizes scheduling costs. First, a task scheduling cost matrix D∈R between expert groups and GPU-accelerators is constructed based on the expert groups (Groups) and the initial expert placement position (Initial_mapping). N×N D ij This represents the scheduling cost incurred when assigning the i-th expert group to the j-th GPU-accelerator, where the j-th GPU-accelerator is denoted as GPU. j If the experts in this group were already working on GPUs... j If the above is executed, no additional scheduling cost is incurred; otherwise, the scheduling cost is accumulated. The formula for calculating the scheduling cost is:

[0045]

[0046] Where |G i | is the number of experts in the i-th group, G i For the i-th expert group, indicator function II is used to determine whether the expert was originally in the GPU. j In the above, Initial_mapping is the initial expert placement position, and the other parameters are those in formula (4). Then, two dynamically adjusted cost arrays are introduced for expert grouping and GPU-accelerator respectively, namely g i and r j These two terms represent the cost concessions that the group and the GPU-accelerator are willing to make in task allocation, respectively, and the allocation of task group i to the GPU is dynamically adjusted in each iteration. j The current cost is calculated using the following formula:

[0047] D′ ij =D ij -g i -r j (9)

[0048] In each iteration, for each expert group, the algorithm selects the device with the lowest cost for allocation based on the cost matrix of the current iteration. After allocation, the system updates the cost array and cost matrix according to the allocation results of this round, reflecting the current load changes and promoting a reduction in overall scheduling cost. This process continues to iterate until the allocation mapping of the expert group tends to stabilize or reaches the preset number of iterations, thereby obtaining a group allocation scheme with a lower overall scheduling cost. Specific details are shown in Algorithm 2.

[0049]

[0050]

[0051] Step four: based on the results of step three, implement the scheduling scheme using an asynchronous communication strategy to improve device resource utilization and reduce communication overhead.

[0052] In order to complete the scheduling of expert parameters and the scheduling of optimizer parameters in the case of expert parallelism, a collective communication operator is needed to exchange parameters between different devices. Based on the new expert placement scheme obtained in step three, compare the original scheme to obtain the parameter position information that needs to be moved on each device, then extract the required parameters from the MoE layer and the optimizer and merge them respectively, then use the AlltoAll communication operator to complete the scheduling of the parameters, and finally place the communication parameters in the corresponding positions.

[0053] For the use of AlltoAll communication, an asynchronous communication strategy is used to implement the scheduling scheme, and the expert parameter scheduling and the optimizer parameter scheduling are placed in the Attention layer calculation and the MoE layer FFN calculation respectively to realize communication scheduling masking. Since there is no dependency between the communication of the former two and the calculation of the latter two, the calculation and communication can be performed simultaneously on the same device through multi-stream, and the long calculation of the two parts can mask the communication time required by the parameter scheduling, improving the utilization of device resources.

[0054] Embodiment:

[0055] The following is a specific application of the method of the present application, an experimental and analysis of the mixed expert model training optimization method based on expert scheduling:

[0056] The hardware environment used in the application is a single-node GPU server and a single-node GPU-accelerator server, and Ubuntu 22.04 operating system builds a complete CUDA software environment to support the training experiment of AI model, and the specific environment settings are shown in Table 1.

[0057] Table 1 Environment Settings

[0058]

[0059] The MoE model of the Mixtral series is used in the experiment of the application, which is divided into two different layers (4 layers and 8 layers), and the model parameter quantity is 6.07B and 11.87B respectively. The GPU server performance comparison data is shown in Tables 2 and 3, and the GPU-accelerator server model performance comparison data is shown in Tables 4 and 5. Since there is a certain relationship between the expert load condition and the data set, the application is tested using multiple data sets, specifically including Alpaca, Wikitext, ShareGPT, and Lambada data sets. In this experiment, the MoE model combined with the Mixtral structure and multiple types of related data sets are used to test the performance of the model training process under different input data volume Gbs, and the end-to-end performance improvement in multiple scenarios is evaluated. Gbs refers to the global input data volume. Base in the test data refers to the training performance under normal EP parallel without using expert scheduling.

[0060] Table 2 GPU server 4-layer Mixtral model performance comparison data

[0061]

[0062] Table 3 GPU server 8-layer Mixtral model performance comparison data

[0063]

[0064] Table 4 GPU-accelerator server 4-layer Mixtral model performance comparison data

[0065]

[0066] Table 5 GPU-accelerator server 8-layer Mixtral model performance comparison data

[0067]

Claims

1. A hybrid expert model training and optimization method based on expert computation load balancing scheduling, characterized in that, Specifically, the following steps are included: Step 1: Predict the global expert load data for the current iteration and obtain sampled data; Step 2: Design a multi-dimensional performance-aware expert placement evaluation method, which transforms multi-dimensional communication and computation into a unified latency index to achieve quantitative comparison and decision-making in the expert placement scheme search stage; Step 3: Based on the sampled data and expert placement evaluation method, an expert placement strategy is designed using a dynamic load feature-aware expert grouping search algorithm and an expert grouping and resource-aware matching algorithm. Step 4: Based on the expert placement strategy, an asynchronous communication strategy is used to schedule expert and optimizer parameters to complete the training optimization.

2. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 1, characterized in that, The specific implementation process of step one is as follows: After the gating network is computed in the hybrid expert model, the expert selection results of each device are summarized by ensemble communication operation, and historical global expert load data and expert fluctuation characteristic data are sampled. The expert fluctuation characteristic data is the standard deviation of historical global expert load data; Design an expert load prediction and evaluation model, using historical global expert load data, and recursively predict the global expert load for the current iteration based on the weighting index α. Where L (t-1) The actual expert load obtained from the previous iteration sampling. The prediction result from the previous round is used, and α is the weighting exponent; an adaptive update mechanism for the weighting exponent is employed to calculate the predicted value after each iteration of the model. Compared with the true value L (t) The error between them, and dynamically adjust α according to the magnitude of the error; The sampling data consists of historical global expert load data, expert fluctuation characteristic data, and predicted global expert load.

3. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 1, characterized in that, The second step is specifically implemented as follows: In terms of benefits, the impact of load on computation latency is modeled using a nonlinear function, and the maximum expert computation load change of each GPU-accelerator before and after expert scheduling is compared to evaluate the scheduling benefits. Communication overhead comes from the scheduling of expert parameters and optimizer parameters; the total scheduling benefit is obtained by subtracting the communication overhead from the scheduling benefit. When the total scheduling benefit exceeds a preset threshold, it is determined that the current round of scheduling needs to be executed.

4. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 3, characterized in that, The specific process of the scheduling benefit is as follows: the actual computing performance of the GPU-accelerator increases non-linearly with the increase of load. A non-linear function f(x) = x·log(1+θx) is used to simulate the impact of load changes on computing time, where θ is an adjustment factor; let Load i Let γ be the load of the i-th device, γ be the computational cost of expert calculations on a single data point, and F be the effective computing power. The scheduling benefit is expressed as: These represent the load on the equipment before and after expert scheduling, respectively.

5. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 3, characterized in that, The scheduling overhead includes expert parameter scheduling overhead and optimizer parameter scheduling overhead, as detailed below: Expert parameter scheduling overhead: Let N be the number of experts that need to be scheduled on the i-th device. i If the expert parameter size is Shape, the data type size is Size, the communication bandwidth is H, and the communication start delay is τ, then the expert parameter-based scheduled communication is represented as: Considering the overlap between expert scheduling communication and Attention calculation, the expert parameter scheduling overhead is expressed as: Where seq is the sequence length of the Attention layer, κ is the unit time consumption coefficient, and ε is a stabilizing term to prevent the denominator from being zero; Optimizer parameter scheduling overhead: The optimizer needs to schedule the mean and variance corresponding to the gradient of each expert parameter. Optimizer parameter scheduling communication... comm Equals 2 * Expert comm The optimizer scheduling communication overlaps with the FFN layer computation of MoE, and the optimizer scheduling overhead is expressed as:

6. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 3, characterized in that, Step three is specifically implemented as follows: First, based on the sampled data, an expert grouping search algorithm based on dynamic load feature perception is used to group the experts, and a load-balanced expert fusion grouping is achieved. Second, a matching algorithm based on expert grouping and resource perception is used to achieve optimal matching between expert groups and GPU-accelerator, so as to minimize the total scheduling cost of the expert placement scheme and finally obtain the scheme with the greatest overall benefit.

7. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 6, characterized in that, The specific implementation process of the expert group search algorithm based on dynamic load feature perception is as follows: First, initialize several empty expert groups, place experts one by one in order of increasing total cost, and record the current cumulative total cost of each group. The calculation formula is P = e + μ·σ, where e and σ are the predicted global expert load and the total value of expert fluctuation characteristics in the group, and μ is the fitting parameter. Based on the sampled data, while keeping the final number of experts in different groups consistent, experts are placed one by one into the group with the lowest overall cost. This cost takes into account both the current total load and the cumulative fluctuation of the group. Finally, an expert combination scheme with small load differences between groups is obtained, and the revenue is calculated according to the expert placement evaluation method in step two.

8. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 6, characterized in that, The matching algorithm based on expert grouping and resource awareness is specifically implemented as follows: First, a scheduling cost matrix is ​​constructed according to the expert placement evaluation method and expert grouping, and the cost array of expert grouping and GPU-accelerator is initialized to represent the cost concession accepted during the allocation process; Subsequently, in each iteration, for each expert group, based on the cost matrix of the current iteration, the device with the lowest current cost is selected for allocation attempt. After the allocation is completed, the cost array and cost matrix are updated according to the allocation result of this round. This process is repeated until the mapping relationship of the expert groups tends to be stable or reaches the preset iteration limit, and finally realizes the mapping scheme with the minimum scheduling overhead between the expert group and the GPU-accelerator.

9. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to claim 8, characterized in that, The cost matrix is ​​specifically implemented as follows: First, a task scheduling cost matrix D∈R between the expert groups and the GPU-accelerator is constructed based on the expert groups (Groups) and the initial expert placement position (Initial_mapping). N×N , where element D ij This represents the scheduling cost incurred when assigning the i-th expert group to the j-th GPU-accelerator, where the j-th GPU-accelerator is denoted as GPU. j If the experts in this group were originally in GPU... j If the above is executed, no additional scheduling cost is incurred; otherwise, the scheduling cost is accumulated. The formula for calculating the scheduling cost is: Where |G i | is the number of experts in the i-th group, G i For the i-th expert group, indicator function II is used to determine whether the expert was originally in the GPU. j Then, two dynamically adjusted cost arrays are introduced for expert grouping and GPU-accelerator respectively, namely g i and r j These two terms represent the cost concessions that the group and the GPU-accelerator are willing to make in task allocation, respectively, and the allocation of task group i to the GPU is dynamically adjusted in each iteration. j The current cost is calculated using the following formula: D′ ij =D ij -g i -r j In each iteration, for each expert group, based on the cost matrix of the current iteration, the device with the lowest cost is selected for allocation. After allocation is completed, the cost array and cost matrix are updated based on the allocation results of this round, and the current load changes are reflected.

10. The hybrid expert model training and optimization method based on expert computation load balancing scheduling according to any one of claims 1 to 9, characterized in that, Step four is specifically implemented as follows: by comparing the old and new expert placement strategies, the scheduling parameters are extracted, and the AlltoAll communication operation is used to perform inter-device scheduling. The scheduling process adopts an asynchronous communication strategy. In the hybrid expert model, the Attention layer and FFN layer respectively embed the scheduling process of expert and optimizer parameters.

Images