Hybrid expert model training method and system for adaptive load balancing

Abstract

The invention relates to an adaptive load balancing hybrid expert model training method and system. The method comprises the following steps: calculating the affinity of each token to each expert; in the first stage, an expert with the highest affinity degree is allocated to each token, discarding of the over-limit tokens is limited according to the capacity of the experts, and the token discarding rate and the affinity degree index of the current batch are calculated; in the second stage, the remaining experts are distributed to each token under the condition that capacity limitation is met through multi-round iterative distribution; and dynamically adjusting the numerical value of the next training batch based on the discarding rate of the current batch and the affinity index. By adopting the method, self-adaptive load balancing can be realized in the pre-training process of the hybrid expert model, and the training efficiency and the resource utilization rate are effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an adaptive load balancing hybrid expert model training method and system. Background Technology

[0002] Deep learning-based neural network technology has developed rapidly in recent years in fields such as computer vision and natural language processing. Recently, large-scale models based on the Transformer structure have achieved remarkable results, demonstrating enormous development potential. As the scale of large models continues to increase, the required computing power also increases. Hybrid expert structures, as a method to increase the scale of model parameters while controlling the computational load, have attracted much attention.

[0003] Popular large-scale model series such as Mixtral, DeepSeek, and Qwen have all introduced hybrid expert architecture models. The hybrid expert layer uses a routing module to calculate the affinity of each token to various experts and selects the experts with the highest affinity to perform computation on the token. In the pre-training of large-scale models with hybrid expert architectures, expert parallelism, as a method of distributing experts across different computing devices to achieve parallel computation, has been widely used.

[0004] Mixture of Experts (MoE) models, due to their sparse activation characteristics, can lead to a few experts processing the majority of tokens while others are underutilized if left unchecked. This wastes network parameters, reduces training effectiveness, and causes training instability. Furthermore, the different execution times of tasks across different devices can cause waiting, reducing hardware resource utilization. To address this, additional auxiliary losses, optimized allocation algorithms, and expert transfer techniques are typically used to balance the load on each expert and device. Summary of the Invention

[0005] Therefore, it is necessary to provide an adaptive load balancing hybrid expert model training method and system to address the aforementioned technical problems.

[0006] Firstly, this application provides a training method for an adaptive load balancing hybrid expert model, including:

[0007] Within the current training batch, calculate the affinity of each token to each expert in the hybrid expert model, where the total number of experts to be selected for each token is K, and the affinity is the probability that each token is assigned to each expert for processing;

[0008] In the first phase of expert allocation, each token was assigned the one with the highest affinity. For each expert, discard any tokens exceeding the capacity limit assigned to them, calculate the token discard rate and affinity index for the current batch, where the affinity index is the highest affinity among all tokens in the current batch. The sum of normalized affinities of all experts;

[0009] In the second phase of expert allocation, through multiple rounds of iterative allocation, under the condition of satisfying the capacity limit, an expert is assigned to each token in each round of iterative allocation until each token is allocated an expert. One expert;

[0010] Based on the token discard rate and affinity metric of the current batch, dynamically adjust the tokens in the next training batch. The values ​​are used to adaptively balance the load of the experts.

[0011] In one embodiment, the assignment of each token to the one with the highest affinity is... For each expert, discard tokens exceeding the capacity limit assigned to that expert, and calculate the token discard rate and affinity metric for the current batch, including:

[0012] Assign the token with the highest affinity to it. Each expert is identified, and the token-explored pair with the expert is marked as explored.

[0013] For each expert, if the number of tokens allocated to them exceeds the capacity limit, then the tokens with the lowest affinity are discarded until the capacity limit is met.

[0014] Calculate the token discard rate and affinity index for the current batch. The formula for calculating the affinity index is as follows:

[0015] ;

[0016] in, Let be the affinity sequence of the i-th token for all experts. Let i be the normalized affinity between the i-th token and the j-th expert. The one with the highest affinity for the i-th token There is a set of experts, and m is the number of tokens.

[0017] In one embodiment, in the expert allocation of the second phase, each round of allocation includes:

[0018] At the start of each allocation round, each token is initialized to an unallocated state for the current round;

[0019] In each round of allocation, multiple explorations are performed. In each exploration, for a token in the unallocated state, the token with the highest affinity is allocated from among its unexplored experts and marked as explored.

[0020] After each exploration is completed, the expert capacity is checked. For experts whose allocated tokens exceed the capacity limit, newly allocated low affinity tokens in this exploration are discarded until the capacity limit is met.

[0021] The tokens discarded in this exploration are restored to an unassigned state and continue to participate in the next exploration until all tokens are assigned to an expert or no more tokens can be assigned.

[0022] In one embodiment, the token discard rate and affinity metric of the current batch are used to dynamically adjust the tokens in the next training batch. The values, used to adaptively balance the expert's load, include:

[0023] Using the moving average prediction method, based on the token discard rate and affinity index of the current batch, the token discard rate and affinity index of the next batch are predicted, and a discriminant function is calculated based on the predicted values.

[0024] If the result of the discrimination function is higher than the preset threshold, then reduce... ;

[0025] If the result of the discrimination function is lower than the preset threshold, then increase .

[0026] In one embodiment, the step of predicting the token discard rate and affinity index for the next batch using a moving average prediction method, based on the token discard rate and affinity index of the current batch, includes:

[0027] The affinity index for the next batch is predicted using the following formula:

[0028] ;

[0029] The discard rate of the next batch of tokens is predicted using the following formula:

[0030] ;

[0031] in, This is the affinity index predicted for the current batch. The affinity index is the actual value calculated for the current batch. This represents the predicted token discard rate for the current batch. is the actual token discard rate calculated for the current batch, and w is the moving average weight.

[0032] In one embodiment, the formula for calculating the discriminant function is:

[0033] ;

[0034] in, The weighting coefficient for affinity index This is the discard rate weighting coefficient, and , For the affinity index of the next batch, This represents the token discard rate for the next batch.

[0035] In one embodiment, the formula for calculating the token discard rate actually obtained in the current batch is:

[0036] ;

[0037] Where d is the number of tokens discarded in the expert allocation of the first stage in the current batch, and m is the total number of tokens.

[0038] In one embodiment, the method for calculating the capacity limit includes:

[0039] For the total number of experts In a hybrid expert model, the number of tokens is m, and the total number of experts that each token needs to select is K. Therefore, the capacity limit for each expert is... .

[0040] Secondly, this application also provides an adaptive load-balanced hybrid expert model training system, comprising:

[0041] The affinity calculation module is used to calculate the affinity of each token to each expert in the hybrid expert model within the current training batch, wherein the total number of experts to be selected for each token is K, and the affinity is the probability that each token is assigned to each expert for processing;

[0042] The first allocation module is used to assign the token with the highest affinity during the first stage of expert allocation. For each expert, discard any tokens exceeding the capacity limit assigned to them, and calculate the token discard rate and affinity index for the current batch. The affinity index is the percentage of tokens that can be discarded within the current batch. The sum of normalized affinities of all experts;

[0043] The second allocation module is used in the second stage of expert allocation to allocate one expert to each token in each iterative allocation round, under the condition of satisfying the capacity limit, through multiple rounds of allocation, until each token is allocated an expert. One expert;

[0044] The parameter adjustment module is used to dynamically adjust the parameters in the next training batch based on the token discard rate and affinity index of the current batch. The values ​​are used to adaptively balance the load of the experts.

[0045] Thirdly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0046] The aforementioned adaptive load-balancing hybrid expert model training method and system utilizes an adaptive two-stage routing approach to achieve load-balanced expert allocation for the hybrid expert module. While ensuring that the load on each expert does not exceed its processing limit, it allocates K experts to each token, preventing a few experts from being overloaded while the majority remain idle. Dynamic adjustment is achieved through a moving average prediction method based on historical drop rates and the sum of affinity scores. This allows the system to strike a balance between efficiency and accuracy. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a diagram illustrating the application environment of a hybrid expert model training method for adaptive load balancing in one embodiment.

[0049] Figure 2 This is a flowchart illustrating a hybrid expert model training method for adaptive load balancing in one embodiment.

[0050] Figure 3 Allocate the remaining amount in one embodiment A flowchart illustrating the expert's steps;

[0051] Figure 4 A flowchart illustrating the training method for a hybrid expert model for adaptive load balancing in a most detailed embodiment.

[0052] Figure 5 This is a block diagram of an adaptive load balancing hybrid expert model training system in one embodiment.

[0053] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The adaptive load balancing hybrid expert model training method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network, submitting model training tasks and initial data. Server 104 deploys a hybrid expert large language model containing multiple experts and is capable of model training. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located on a cloud or other network server. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0056] In one exemplary embodiment, such as Figure 2 As shown, an adaptive load balancing hybrid expert model training method is provided, which is then applied to... Figure 1 The training process of a large language model based on a hybrid expert model deployed on a server is used as an example for illustration, including the following steps S201 to S204. Wherein:

[0057] Step S201: Within the current training batch, calculate the affinity of each token for each expert in the hybrid expert model.

[0058] Wherein, the total number of experts to be selected for each token is K, and the affinity is the probability that each token is assigned to each expert for processing.

[0059] It should be noted that this method is designed based on the batch training iteration process of MoE pre-training. In the training of MoE, the model iteratively learns through multiple training batches (batches). At the beginning of each training batch, a batch of tokens is input into the model. Among them, the routing module is the scheduling module in the expert layer of the model, which can receive the input token sequence, calculate the affinity between each token and all experts, screen out the experts that best match each token based on the affinity, and at the same time coordinate the expert load to avoid resource waste or overload. It can be understood that if the routing module is not designed reasonably, for example, only allocated according to a fixed Top-K, there will be a situation where a small number of experts process most of the tokens and the rest are idle, resulting in unstable training and low hardware utilization rate.

[0060] The routing module usually consists of a linear transformation, and its linear layer parameters can be learned and optimized through gradient descent during model training. In a network with n experts, for an input of m tokens, after passing through the linear layer of the routing module, a matrix is output , where represents the affinity of the i-th token to the j-th expert. This matrix is the basis for making subsequent expert allocation decisions within this batch.

[0061] Step S202, the first-stage expert allocation. In the first-stage expert allocation, the experts with the highest affinity are allocated to each token, and the tokens that exceed the capacity limit assigned to each expert are discarded, and the token discard rate and affinity index of the current batch are calculated.

[0062] Among them, the affinity index is the sum of the normalized affinities of the experts with the highest affinity within the current batch. Among them, the expert capacity limit defines the maximum number of tokens that a single expert can process in one forward propagation. Its calculation is based on the following parameters of the model: the total number of experts n, the number of input tokens m, and the total number of experts K that each token needs to select. The above parameters are hyperparameters for model training and are determined before training. Each input token will select K out of n experts for calculation, where K < n. In practical applications, the above parameter values can usually be adjusted so that the calculation result of the expert capacity limit is an integer. Exemplarily, the expert capacity limit can be set as: .

[0063] This method divides the expert allocation into two stages. In the first stage, a Selection and capacity constraints are used to achieve rapid expert allocation. In the second phase, the allocation method is explored iteratively to replenish the remaining expert quota for each token within the capacity limit. In both phases, by setting... The value comes from the accuracy and balance assigned by the adaptive adjustment experts.

[0064] This step is the first phase. Specifically, it involves assigning each token the token with the highest affinity. Each expert is assigned a number of tokens, and the allocation results are recorded. Subsequently, for any expert whose allocated token count exceeds their capacity limit, tokens with the lowest affinity are discarded, starting with those with the lowest affinity, until the capacity limit is met. Simultaneously, for each token, its... Affinity experts marked the assignment as explored and recorded the discard rate during this allocation. and Normalized Affinity Sum of Experts .

[0065] Among them, the token discard rate This refers to the proportion of tokens discarded due to expert restrictions in the current global batch, out of the total number of tokens.

[0066] ;

[0067] Where d is the number of tokens discarded, and m is the number of input tokens.

[0068] The affinity sum is calculated based on the total affinity of all tokens in the current global batch within their corresponding... The sum of normalized affinity values ​​on the expert set is:

[0069] ;

[0070] Where P is the affinity matrix. Let be the affinity value of the i-th token with all experts. Let be the normalized affinity value of the i-th token to the j-th expert. The token with the highest affinity for the i-th token There is a set of experts, and m is the number of tokens.

[0071] It's understandable that the token drop rate reflects load balancing, while the normalized affinity sum reflects route confidence. When the load is severely unbalanced, a few high-affinity experts will be assigned a large number of tokens. To meet expert capacity limitations, a large number of low-affinity tokens assigned to these experts need to be dropped, leading to a high drop rate. Therefore, a low drop rate reflects that the current allocation strategy maintains load balancing. Route confidence is used to evaluate the reliability of the routing module's allocation decisions. When route confidence is high, for most tokens, the routing module can determine that a few experts are significantly better than others. This makes each token's... The normalized affinity scores of individual experts are highly concentrated in a few values ​​(i.e., close to 1), while other values ​​are close to 0. Summing these high-confidence values ​​yields a high total affinity score, and vice versa. Therefore, the total normalized affinity score can reflect the confidence level of the current routing allocation strategy. This invention monitors these two dimensions simultaneously to inform subsequent adaptive adjustments. Provide complementary evidence.

[0072] Step S203, Second Stage Expert Allocation. In the second stage of expert allocation, through multiple rounds of iterative allocation, under the condition of satisfying the capacity limit, one expert is assigned to each token in each round of iterative allocation until each token is allocated an expert. One expert.

[0073] This step is the second allocation phase. It can be understood that each token has already been allocated in step S202 above. According to the initial rules, each token should be allocated K tokens, therefore, further allocation is needed. K experts. It should be noted that tokens discarded due to expert capacity limitations in the first phase of allocation should still be considered as having selected K experts and will directly enter the second phase of expert allocation.

[0074] Specifically, this step is achieved through iterative allocation. The goal of each round of allocation is to assign a target expert with relatively high affinity to each token in each allocation. The iteration is performed in total. Rounds. In each round of allocation, each token will be attempted to be allocated from its unexplored experts, and the token-expert pair will be marked as explored after being allocated to an expert.

[0075] For example, a multi-exploration mechanism can be employed within each round of allocation. In each exploration, for tokens that have not yet been allocated to an expert in this round, the expert with the highest affinity among the unexplored experts is selected for an attempt to allocate the token. After a single exploration, the corresponding expert capacity is checked, and tokens for experts exceeding the capacity limit are discarded.

[0076] After expert allocation of all tokens is completed, an auxiliary loss is applied to the overall allocation results for parameter optimization of the routing module. The auxiliary loss and the model's main loss function together constitute the model's training objective.

[0077] Step S204: Based on the token discard rate and affinity metrics of the current batch, dynamically adjust the tokens in the next training batch. The value is used to adaptively balance the load of experts.

[0078] After completing the computation of a global batch, in the gradient aggregation phase, the token drop rate and normalized affinity sum recorded in this batch are simultaneously aggregated. Combined with data from a certain number of previous global batches, the token drop rate and normalized affinity sum of the next global batch are predicted. For example, the moving average prediction method can form an estimate of the future by assigning decaying weights to historical data. In this step, a moving average method based on an exponentially weighted moving average model can be used for prediction.

[0079] Then, based on the calculated token discard rate prediction and the normalized affinity sum prediction, the calculation is used to evaluate... The discriminant function for the effect, and adjustments based on the discriminant function results. For example, when the system load pressure increases, it can be increased. This expands the pool of expert token selections, while reducing the threshold when route confidence is high and load is stable. To improve training efficiency.

[0080] also, The adjustment strategy can vary at different stages of training. Considering that exceeding capacity and resulting in a large number of dropped tokens in the early stages of training could be disastrous, then regarding... The adjustment criteria will relax as training progresses; and as the load gradually converges to equilibrium in the later stages of training, there is no need for frequent adjustments. The interval for making judgments will increase as training progresses.

[0081] It should also be noted that when the above method is used for distributed parallel computing, since each node only generates the token discard rate and affinity sum of the data it processes, these need to be aggregated to the master node to calculate complete prediction data. To calculate the predicted value reflecting the global load status, after completing the forward and backward computation of a global batch, at the gradient aggregation synchronization point, the local indicator data recorded by each node needs to be transmitted to the master node. Based on the aggregated global indicator data, the master node obtains complete data for predicting the state of the next batch. The timing of the above aggregation and prediction steps can be chosen to be any time after the indicator aggregation is completed and before the allocation decision for the next global batch begins.

[0082] The above method breaks down the traditional method of allocating experts based on route calculation into a phased expert allocation process, including the first allocation phase based on Top... The first phase involves rapid filtering, followed by an iterative exploration-based completion phase in the second phase. This segmented mechanism ensures both the efficiency of the initial allocation and the ability to achieve high-quality expert matching for all tokens within the expert capacity constraint. Adjustments are made using the token discard rate and affinity metrics from the first phase. This allows for adjustments to the expert allocation in the first and second stages, enabling adaptive optimization of the routing strategy during training to achieve a balance between routing accuracy and computational efficiency.

[0083] In one exemplary embodiment, each token is assigned the one with the highest affinity to it. For each expert, discard tokens exceeding the training capacity limit of the adaptive load balancing hybrid expert model. Calculate the token discard rate and affinity metric for the current batch of training in the adaptive load balancing hybrid expert model, including the following steps:

[0084] Step S301: Assign each token the one with the highest affinity to it. Each expert is identified, and the matching pair of tokens and experts is marked as explored.

[0085] Step S302: For each expert, if the number of tokens allocated to them exceeds the capacity limit, then the tokens with the lowest affinity are discarded until the capacity limit is met.

[0086] Step S303: Calculate the token discard rate and affinity index of the current batch.

[0087] Specifically, the affinity index for the current batch is calculated as follows:

[0088] ;

[0089] in, Let be the affinity sequence of the i-th token for all experts. Let i be the normalized affinity between the i-th token and the j-th expert. The one with the highest affinity for the i-th token There is a set of experts, and m is the number of tokens.

[0090] In the above embodiment, the initial expert allocation process is completed in the first stage of allocation, and the affinity index and token discard rate of the current batch are recorded. Through a one-time high affinity expert allocation, most of the high matching degree allocation can be completed quickly. By calculating the affinity index and token discard rate, the confidence of the routing decision and the load imbalance of the current batch are quantified, thereby providing a data basis for the decision optimization of the next batch.

[0091] In one exemplary embodiment, such as Figure 3 As shown, in the second phase of expert allocation, each round of allocation includes the following steps:

[0092] Step S401: At the start of each round of allocation, initialize each token to an unallocated state in the current round.

[0093] First, at the start of each allocation round, the allocation state of all tokens is initialized to the unallocated state. For example, in a specific scenario, the total number of experts n=4, the number of tokens m=10, and K=3. =1, expert capacity is limited to 8. The overall task of the second phase is to allocate the remaining 2 experts to each token, so the system will execute two independent allocation processes.

[0094] Assume that after the first phase of allocation, the initial state of the current batch is as shown in Table 1:

[0095] Table 1

[0096]

[0097] At this point, assume that the initial load for each expert is: E1: 4 / 8, E2: 3 / 8, E3: 5 / 8, E4: 2 / 8.

[0098] Step S402: Perform multiple explorations in each round of allocation. In each exploration, for a token with an unallocated state, allocate the token with the highest affinity from its unexplored experts and mark it as explored.

[0099] In the first exploration, the token experts were assigned as follows:

[0100] T1: Select E2 (0.7) with the highest affinity from the unexplored {E2,E3,E4}, and update the explored set {E1,E2};

[0101] T2: Select E1 (0.6) with the highest affinity from the unexplored {E1,E3,E4}, and update the explored set {E1,E2};

[0102] T3: Select E4 (0.7) from the unexplored {E1,E2,E4}, which has the highest affinity, and update the explored set {E3,E4}.

[0103] T4: Select E3 (0.3) from the unexplored {E2,E3,E4} with the highest affinity and update the explored set {E1,E3};

[0104] The other tokens were explored in a similar manner.

[0105] Step S403: After each exploration is completed, check the expert capacity. For experts whose allocated tokens exceed the capacity limit, discard the newly allocated low affinity tokens in this exploration until the capacity limit is met.

[0106] After this exploration is completed, check the capacity limits of all experts who were assigned tokens in this exploration. For experts whose capacity limits are exceeded, discard the tokens with the lowest affinity among all newly assigned tokens to that expert in this exploration, starting with the token with the lowest affinity, until the capacity limit is met. For example, after all the above tokens have been assigned, suppose expert E3 was assigned 9 tokens, exceeding the capacity limit, and T4 is the token with the lowest affinity among all tokens assigned to E3, then T4 will be discarded.

[0107] Step S404: Restore the tokens that were discarded in this exploration to an unassigned state and continue to participate in the next exploration until all tokens are assigned to an expert or no more tokens can be assigned.

[0108] In this step, the discarded T4 will revert from "assigned" to "unassigned" in this round of allocation, but its set of explored experts will remain {E1, E3}. This means that in subsequent explorations, T4 will not choose expert E3 again, but can still choose other experts from its unexplored experts {E2, E4}. The state update of each token at this time is shown in Table 2:

[0109] Table 2

[0110]

[0111] Therefore, in the second exploration, the unassigned T4 at this point selects E2, which has the highest affinity, from {E2, E4}, and the other unassigned tokens are allocated in the same way. After allocation, a capacity check is performed again.

[0112] The process iterates multiple times until all tokens are assigned to an expert or no further assignments are possible.

[0113] Then, the second round of allocation begins, assigning a third expert to each token. In this second round, a similar iterative exploration mechanism is used to finally complete the expert allocation for all tokens.

[0114] In the above embodiments, through an iterative mechanism of multiple rounds of allocation and multiple explorations, after the initial screening in the first stage, the remaining experts can be allocated to each token without repetition, under the constraint of expert capacity limitations. Even if some tokens are discarded in the initial exploration, they can be redistributed to the remaining experts in this stage, thereby improving the success rate and quality of expert allocation.

[0115] In an exemplary embodiment, using the data from the above embodiment as an example, the method of dynamically adjusting the prediction based on the token discard rate and the sum of affinity in historical allocations using a moving average method is explained in detail. The method of numerical values.

[0116] By using the moving average method, combined with historical token discard rates and historical average affinity, the discard rate and affinity indicators of the next batch of tokens are predicted, and a discriminant function is calculated based on the predicted values.

[0117] First, let the current training batch number be t, and the moving average weight be w = 0.9. In the previous training round, the predicted values ​​of the token discards and the sum of normalized affinities for the current batch are... , After the current batch allocation is completed, the actual measured values ​​are calculated as follows:

[0118] The token discard rate is:

[0119] ;

[0120] Where d is the number of tokens discarded in the first round of allocation, and m is the number of input tokens;

[0121] The affinity index is:

[0122] ;

[0123] Based on the above formula, the actual measured values ​​for the current batch are calculated as follows: , .

[0124] In some embodiments, for example, the token discard rate and affinity index of the next batch are predicted based on the token discard rate and affinity index of the current batch using a moving average prediction method.

[0125] Specifically, the affinity index for the next batch is predicted using the following formula:

[0126] ;

[0127] The discard rate of the next batch of tokens is predicted using the following formula:

[0128] ;

[0129] in, This is the affinity index predicted for the current batch. The affinity index is the actual value calculated for the current batch. This represents the predicted token discard rate for the current batch. is the actual token discard rate calculated for the current batch, and w is the moving average weight.

[0130] The predicted affinity index value for the next batch is:

[0131] ;

[0132] The predicted token discard rate for the next batch is:

[0133] .

[0134] Based on the two indicators for the next batch, a discriminant function can be calculated to predict the values. In an exemplary embodiment, the discriminant function is defined as:

[0135] ;

[0136] in, The weighting coefficient for affinity index This is the discard rate weighting coefficient, and , For the next batch of affinity indicators, This is the token discard rate for the next batch.

[0137] Specifically, in the parameter settings, the threshold of the discriminant function is set to... The weights of the discriminant function are , According to the above formula, the value of the discriminant function is:

[0138] .

[0139] Compare the discriminant function value with the preset threshold. In comparison, due to This indicates that the current route confidence is insufficient and the load pressure is high. Therefore, [the route should be adjusted accordingly]. Increase from 1 to 2. (Increase) Subsequently, the first phase of the next training batch will expand the range of experts to be selected, allowing more tokens to be assigned to suitable experts in the first phase, reducing the token drop rate, and reducing the iteration process in the second phase, thereby ensuring representational capability by retaining high-affinity expert combinations.

[0140] By using the above-mentioned moving average prediction method, the routing optimization strategy can be adaptively adjusted according to the feedback during the training process, thereby improving training efficiency while ensuring load balance and thus achieving resource allocation optimization.

[0141] Based on the above embodiments The discussion of the flexibility of the adjustment phase, in a preferred embodiment, will... The adjustment phase is placed after the preliminary calculations and before the route affinity calculations, and will be described in detail in conjunction with other embodiments described above. Its flowchart is as follows: Figure 4 As shown, it includes the following steps:

[0142] Step S501, preliminary calculation.

[0143] The system first performs preprocessing calculations, preprocessing and extracting features from the input token to prepare for subsequent route calculations and expert allocation.

[0144] Step S502, Adjustment.

[0145] Based on the dynamic load situation, adjust the parameters. Make dynamic adjustments.

[0146] Step S503, route calculation.

[0147] Based on the features of the input tokens from the preceding computation, routing computation is performed to generate the affinity between each token and each expert.

[0148] Step S504, First Stage Top Experts assigned.

[0149] Based on affinity scores, each token is assigned the token with the highest affinity score from the first phase. One expert.

[0150] Step S505: Check if the expert capacity is exceeded.

[0151] Check if the current allocation of all experts exceeds their preset capacity limit. If not, proceed to S506; if so, proceed to S508.

[0152] Step S506: Record the token discard rate and affinity metrics.

[0153] If the limits are not exceeded, record the current discard status of each token and related affinity metrics for subsequent performance analysis and optimization.

[0154] Step S507, let n = +1.

[0155] Initialize the counter n for the second round of allocation to... +1 indicates that the search for the token has begun. +1 expert.

[0156] Step S508: Discard the token with the lowest affinity among the over-limit experts.

[0157] If the current allocation causes the expert capacity to exceed the limit, the system will discard the token with the lowest affinity among the experts who exceed the limit in order to release capacity and optimize the allocation.

[0158] Step S509: Explore the remaining highest affinity experts for each token that has not yet obtained the nth expert.

[0159] For tokens that have not yet been assigned to the nth expert, the system continues to explore the candidate expert with the highest affinity among its remaining unassigned experts.

[0160] Step S510: Determine if the expert capacity exceeds the limit.

[0161] Check again whether the expert capacity exceeds the limit. If it does not exceed the limit, proceed to S511; if it exceeds the limit, perform the S512 discard operation.

[0162] Step S511: Determine whether all tokens have been assigned to the nth expert or there are no other possible assignments.

[0163] The system determines whether all tokens have already been assigned to the nth expert, or whether there are no more experts available for allocation. If yes, proceed to step S513; otherwise, return to step S509 to continue allocation.

[0164] Step S512: Discard the token with the lowest affinity among the over-limit experts.

[0165] If the allocation in the current nth round causes the expert capacity to exceed the limit, then among the experts who exceed the limit in the current nth round, discard the token with the lowest affinity allocated in the current nth round to release capacity and optimize the allocation.

[0166] Step S513: Determine whether n is equal to K.

[0167] Check if the current allocation stage n has reached the preset maximum number of expert allocations K. If it has, proceed to S515; otherwise, proceed to S514.

[0168] Step S514, let n = n+1.

[0169] The counter n increments during the allocation phase, indicating the start of the next round of expert allocation in the second phase.

[0170] Step S515: Calculate the application auxiliary loss.

[0171] After all tokens complete the second phase After assigning experts, auxiliary loss functions are calculated and applied to optimize routing strategies and affinity learning.

[0172] Step S516, subsequent calculations.

[0173] After expert assignment and loss calculation are completed, the subsequent calculation stage begins, including forward reasoning by each expert and feature fusion.

[0174] Step S517: Collect gradients and simultaneously collect the discard rate and affinity metrics.

[0175] During the backpropagation process, gradients from various expert modules are aggregated, and token discard rates and affinity metrics at each stage are summarized for overall model updates and monitoring.

[0176] Step S518, gradient descent.

[0177] Based on the aggregated gradients and losses, the system executes the gradient descent algorithm to update the model parameters and complete one training iteration.

[0178] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0179] Based on the same inventive concept, this application also provides an adaptive load balancing hybrid expert model training system for implementing the methods described above. The solution provided by this system is similar to the implementation described in the above methods; therefore, the specific limitations in one or more embodiments provided below can be found in the limitations of the adaptive load balancing hybrid expert model training method described above, and will not be repeated here.

[0180] In one exemplary embodiment, such as Figure 5 As shown, an adaptive load-balanced hybrid expert model training system 600 is provided, comprising:

[0181] The affinity calculation module 601 is used to calculate the affinity of each token to each expert in the hybrid expert model within the current training batch, wherein the total number of experts to be selected for each token is K, and the affinity is the probability that each token is assigned to each expert for processing;

[0182] The first allocation module 602 is used to allocate each token with the highest affinity during the first stage of expert allocation. For each expert, discard any tokens exceeding the capacity limit assigned to them, and calculate the token discard rate and affinity index for the current batch. The affinity index is the percentage of tokens that can be discarded within the current batch. The sum of normalized affinities of all experts;

[0183] The second allocation module 603 is used in the second stage of expert allocation to allocate one expert to each token in each iterative allocation round, under the condition of satisfying the capacity limit, through multiple rounds of allocation, until each token is allocated an expert. One expert;

[0184] Parameter adjustment module 604 is used to dynamically adjust the parameters in the next training batch based on the token discard rate and affinity index of the current batch. The values ​​are used to adaptively balance the load of the experts.

[0185] The modules in the aforementioned adaptive load balancing hybrid expert model training system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0186] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores a hybrid expert large language model and data used for model training. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an adaptive load-balanced hybrid expert model training method.

[0187] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0191] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for an adaptive load balancing hybrid expert model, characterized in that, The method includes: Within the current training batch, calculate the affinity of each token to each expert in the hybrid expert model, where the total number of experts to be selected for each token is K, and the affinity is the probability that each token is assigned to each expert for processing; In the first phase of expert allocation, each token was assigned the one with the highest affinity. For each expert, discard any tokens exceeding the capacity limit assigned to them, calculate the token discard rate and affinity index for the current batch, where the affinity index is the highest affinity among all tokens in the current batch. The sum of normalized affinities of all experts; In the second phase of expert allocation, through multiple rounds of iterative allocation, under the condition of satisfying the capacity limit, an expert is assigned to each token in each round of iterative allocation until each token is allocated an expert. One expert; Based on the token discard rate and affinity metric of the current batch, dynamically adjust the tokens in the next training batch. The value is used to adaptively balance the load of the experts.

2. The method according to claim 1, characterized in that, The token is assigned the one with the highest affinity to it. For each expert, discard tokens exceeding the capacity limit assigned to that expert, and calculate the token discard rate and affinity metric for the current batch, including: Assign the token with the highest affinity to it. Each expert is identified, and the token-explored pair with the expert is marked as explored. For each expert, if the number of tokens allocated to them exceeds the capacity limit, then the tokens with the lowest affinity are discarded until the capacity limit is met. Calculate the token discard rate and affinity index for the current batch. The formula for calculating the affinity index is as follows: ； in, Let be the affinity sequence of the i-th token for all experts. Let i be the normalized affinity between the i-th token and the j-th expert. The one with the highest affinity for the i-th token There is a set of experts, and m is the number of tokens.

3. The method according to claim 2, characterized in that, In the second phase of expert allocation, each round of allocation includes: At the start of each allocation round, each token is initialized to an unallocated state for the current round; In each round of allocation, multiple explorations are performed. In each exploration, for a token in the unallocated state, the token with the highest affinity is allocated from among its unexplored experts and marked as explored. After each exploration is completed, the expert capacity is checked. For experts whose allocated tokens exceed the capacity limit, newly allocated low affinity tokens in this exploration are discarded until the capacity limit is met. The tokens discarded in this exploration are restored to an unassigned state and continue to participate in the next exploration until all tokens are assigned to an expert or no more tokens can be assigned.

4. The method according to claim 1, characterized in that, The token discard rate and affinity metric of the current batch are used to dynamically adjust the tokens in the next training batch. The values, used to adaptively balance the expert's load, include: Using the moving average prediction method, based on the token discard rate and affinity index of the current batch, the token discard rate and affinity index of the next batch are predicted, and a discriminant function is calculated based on the predicted values. If the result of the discrimination function is higher than the preset threshold, then reduce... ; If the result of the discrimination function is lower than the preset threshold, then increase .

5. The method according to claim 4, characterized in that, The step of predicting the token discard rate and affinity index for the next batch using the moving average prediction method, based on the token discard rate and affinity index of the current batch, includes: The affinity index for the next batch is predicted using the following formula: ； The discard rate of the next batch of tokens is predicted using the following formula: ； in, This is the affinity index predicted for the current batch. The affinity index is the actual value calculated for the current batch. This represents the predicted token discard rate for the current batch. is the actual token discard rate calculated for the current batch, and w is the moving average weight.

6. The method according to claim 4, characterized in that, The formula for calculating the discriminant function is as follows: ； in, The weighting coefficient for affinity index This is the discard rate weighting coefficient, and , For the affinity index of the next batch, This represents the token discard rate for the next batch.

7. The method according to claim 5, characterized in that, The formula for calculating the token discard rate actually obtained in the current batch is as follows: ； Where d is the number of tokens discarded in the expert allocation of the first stage in the current batch, and m is the total number of tokens.

8. The method according to claim 1, characterized in that, The method for calculating the capacity limit includes: For the total number of experts In a hybrid expert model, the number of tokens is m, and the total number of experts that each token needs to select is K. Therefore, the capacity limit for each expert is... .

9. An adaptive load-balanced hybrid expert model training system, characterized in that, The system includes: The affinity calculation module is used to calculate the affinity of each token to each expert in the hybrid expert model within the current training batch, wherein the total number of experts to be selected for each token is K, and the affinity is the probability that each token is assigned to each expert for processing; The first allocation module is used to assign the token with the highest affinity during the first stage of expert allocation. For each expert, discard any tokens exceeding the capacity limit assigned to them, and calculate the token discard rate and affinity index for the current batch. The affinity index is the percentage of tokens that can be discarded within the current batch. The sum of normalized affinities of all experts; The second allocation module is used in the second stage of expert allocation to allocate one expert to each token in each iterative allocation round, under the condition of satisfying the capacity limit, through multiple rounds of allocation, until each token is allocated an expert. One expert; The parameter adjustment module is used to dynamically adjust the parameters in the next training batch based on the token discard rate and affinity index of the current batch. The values ​​are used to adaptively balance the load of the experts.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images