Load balancing training method and device for mixed expert model

Abstract

This application discloses a load-balanced training method and apparatus for a hybrid expert model, relating to the field of data processing technology. The method includes acquiring training data and training steps; inputting input samples into the routing module of the hybrid expert model to obtain the allocation probabilities corresponding to the expert modules output by the routing module; determining the corresponding target output result and the load of the expert modules based on the allocation probabilities; determining the main loss value based on the target output result and output samples in the training data; determining the balancing loss value based on the load corresponding to the expert modules; determining the total loss value based on the training steps, the balancing loss value, and the main loss value; and optimizing the target expert module and the routing module using the total loss value. By employing load balancing for expert model training, the accuracy of the output is guaranteed, expert module overload or idleness is avoided, communication costs and training cycles are reduced, and the utilization rate of computing resources and the operating efficiency of nodes are increased.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a load balancing training method and device for hybrid expert models. Background Technology

[0002] In related technologies, hybrid expert models allocate tasks to multiple expert modules and combine this with a dynamic allocation mechanism of a routing network for lexical assignment. However, current lexical assignment suffers from imbalance, leading to overload of some expert modules while others remain idle. During model training, expert modules may be distributed across multiple computing nodes, and this uneven load increases communication costs between nodes, reducing computing power utilization and system node operating efficiency. Summary of the Invention

[0003] This application provides a load-balanced training method and device for hybrid expert models, which at least solves the problem in related technologies that during the training process of expert models, expert modules may be distributed across multiple computing nodes, and uneven load will lead to increased communication costs between nodes and extended training cycles.

[0004] This application provides a consultation response method, comprising: acquiring training data and training steps, wherein the training data includes input samples and output samples; converting the input samples into tokens; inputting the tokens into the routing module of a hybrid expert model to obtain the expert assignment probability corresponding to the expert module output by the routing module; determining the load corresponding to the expert module based on the expert assignment probability; determining the target expert module based on the expert assignment probability; inputting the tokens into the target expert module to obtain the target output result output by the target expert module; determining the main loss value based on the target output result and the output samples; determining the equilibrium loss value based on the load corresponding to the expert module; determining the total loss value based on the training steps, the equilibrium loss value, and the main loss value; and optimizing the target expert module and the routing module using the total loss value.

[0005] This application also provides a consultation response device, comprising: a data acquisition module for acquiring training data and training steps, wherein the training data includes input samples and output samples; a sample conversion module for converting input samples into tokens; a probability acquisition module for inputting tokens into the routing module of a hybrid expert model to obtain the expert assignment probabilities corresponding to the expert modules output by the routing module; a load determination module for determining the load corresponding to the expert modules based on the expert assignment probabilities; a module determination module for determining the target expert module based on the expert assignment probabilities; an information acquisition module for inputting tokens into the target expert module to obtain the target output result output by the target expert module; a main loss determination module for determining the main loss value based on the target output result and output samples; a balance loss determination module for determining the balance loss value based on the load corresponding to the expert modules; a total loss determination module for determining the total loss value based on the training steps, the balance loss value, and the main loss value; and a model correction module for optimizing the target expert module and the routing module using the total loss value.

[0006] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described consultation and response methods.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described consultation and response methods.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described consultation and response methods.

[0009] The load-balanced training method and device for hybrid expert models provided in this application, through the collaborative optimization of main loss and balance loss, enables the routing module of the trained consultation response model to have the ability to dynamically balance word allocation. This ensures the accuracy of consultation response while avoiding overload or idleness of expert modules. At the same time, by setting the termination condition for iterative training, the balance between model training efficiency and effect is ensured, solving the problem of some experts being overloaded and others being idle due to uneven word allocation in traditional hybrid expert model training. It also solves the problem that single main loss training cannot take load balancing into account, which leads to increased communication costs and extended response cycles in multi-computing node scenarios. This improves the utilization rate of computing resources and increases the operating efficiency of system nodes. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of the load balancing training method provided in the embodiments of this application;

[0012] Figure 2 A flowchart illustrating the load balancing training method provided in this application;

[0013] Figure 3 A schematic diagram of the load balancing training device provided in this application;

[0014] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0016] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0017] With the widespread application of large-scale language models (LLM) in fields such as natural language processing and machine learning, the Mixture of Experts (MoE) architecture has become the mainstream architecture for large-scale language models due to its excellent parallelization capabilities and scalability.

[0018] The core logic of the hybrid expert model is to allocate lexical units to multiple expert modules for parallel processing through a dynamic allocation mechanism of the routing network. However, existing lexical unit allocation suffers from imbalance, easily leading to some experts being overloaded while others are idle. Especially when expert modules are distributed across multiple computing nodes, uneven load distribution significantly increases the communication cost between nodes, thereby prolonging the response cycle. To enable those skilled in the art to better understand the solution of this application, the following detailed description is provided in conjunction with the accompanying drawings and specific embodiments.

[0019] This section describes the specific application environment architecture or hardware architecture upon which the consultation response method depends. (References) Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the load balancing training method provided in the embodiments of this application. For example... Figure 1 As shown, the application scenarios of the consultation response method include: database 101 and server 102.

[0020] Database 101 may include a single database or a combination of multiple databases; this application does not impose any special restrictions on this. Database 101 is used to store data used for model training.

[0021] Server 102 may include a single server or a cluster of multiple servers.

[0022] Server 102 is used to read the data required for training from database 101, and to train the response model using the read training data and a load-balanced training method.

[0023] Figure 2 This is a flowchart illustrating the load balancing training method provided in this application. Figure 2 As shown, the load balancing training process includes:

[0024] S200: Acquire training data and training steps, where the training data includes input samples and output samples.

[0025] In this step, training data is read from a preset dataset storage medium (local file, database, cloud storage). Input samples can include at least one of text, images, audio, video, etc., and output samples are the corresponding standard output content. The current training step number is read from the training state management module (initially 0; if training is resumed after an interruption, the step number at the time of the last interruption is read), and a preset training step threshold (the upper limit of the number of steps before training stops) is loaded.

[0026] The input samples can be pre-labeled text, images, videos, audio, prompts, etc., used for training by staff. In one possible implementation, the output samples can be the output results corresponding to the input samples, as labeled by staff. The input samples can include consultation information such as business consultations and technical consultations, and the corresponding output samples can include the corresponding consultation results.

[0027] S201: Convert the input sample into a word.

[0028] In this step, the word segmenter can be called to preprocess the input samples, dividing the input samples into the smallest semantic units (sub-words / words). Finally, through the word segmenter's vocabulary mapping, each semantic unit is converted into a word that the model can recognize. At the same time, special words required by the model (such as start words, separator words, etc.) can also be added.

[0029] S202: Input the lexical units into the routing module of the hybrid expert model to obtain the expert assignment probabilities corresponding to the expert modules output by the routing module.

[0030] In this step, after the word tokens are converted into embedding vector matrices by the model embedding layer, they are input into the fully connected layer of the routing module. The fully connected layer obtains the original score vector by performing matrix multiplication between the weight matrix and the embedding vector matrix and adding bias terms. The softmax activation function is applied to the original score vector to map the scores to probability values ​​in the 0-1 interval, i.e., expert assignment probabilities. Each probability value corresponds to an expert module, and the sum of the probability values ​​of all experts is 1. Finally, the output is the assignment probability matrix of each word token in the batch to each expert.

[0031] S203: Determine the load corresponding to the expert module based on the expert allocation probability.

[0032] In this step, a probability-weighted summation method is used to calculate the load. The expert assignment probabilities for all words within the batch are summed along the expert dimension to obtain the total assignment probability for each expert. Dividing the total assignment probability by the total number of words in the batch yields the average load value for each expert (reflecting the expected frequency of expert being assigned words). Alternatively, a hard-counting method can be used for verification: count the number of times each expert is selected by words (ranked among the top k in assignment probability), divide by the product of the total number of words in the batch and k, and obtain the hard-counted load value.

[0033] S204: Determine the target expert module based on the expert assignment probability.

[0034] In this step, a hyperparameter k (the number of experts selected for each word, k≥1) can be preset to sort the expert allocation probabilities for each word in the batch in descending order. The expert module indices corresponding to the top k probability values ​​after sorting are extracted, and the expert modules corresponding to these indices are determined as the target expert modules for that word. The target expert modules for all words are summarized to generate a subset of words to be processed corresponding to each expert module.

[0035] S205: Input the word elements into the target expert module and obtain the target output result output by the target expert module.

[0036] In this step, the embedding vector of each word (or a subset of the words to be processed in step S204 above) is input into its corresponding target expert module. The expert module performs feature transformation and semantic modeling on the embedding vector through its own network structure. Each target expert module outputs the semantic feature vector of that word. The feature vectors output by all target expert modules are weighted and summed according to their assigned probabilities to obtain the fused feature vector of the word. After the fused feature vectors of a batch of words are summarized, they are converted into the target output result through an intermediate fully connected layer.

[0037] The semantic modeling methods include matrix multiplication, activation function operations, and normalization. The target output can be in the form of a feature vector, which is subsequently used for loss calculation.

[0038] S206: Determine the main loss value based on the target output and the output samples.

[0039] In this step, the output sample (standard response text) is first converted into a sequence of integer words (true labels) using a word segmenter. The target output (preliminary feature representation) is then input into the model's output layer. A fully connected layer maps the feature dimensions to the vocabulary size, and a softmax activation function is used to generate the probability distribution of predicted words. The cross-entropy loss function is used to calculate the error between the predicted probability distribution and the true labels. Specifically, the negative logarithm of the predicted probability at each position is taken, summed according to the position of the true label, and then divided by the batch size to obtain the average main loss value, which quantifies the model's prediction bias on the main task.

[0040] S207: Determine the balancing loss value based on the load corresponding to the expert module.

[0041] In this step, a preset load balancing loss calculation method (such as mean squared error loss) is used. First, the average load value of all expert modules is calculated. Then, the difference between each expert's load value and the average load value is calculated, and the difference is squared. The sum of all squared results is divided by the number of expert modules to obtain the mean squared error loss value (i.e., the load balancing loss value). If Shannon entropy is used to calculate the load balancing loss value, the negative entropy of the probability distribution of each expert's load is calculated and used as the load balancing loss value, thus quantifying the degree of load imbalance among expert modules.

[0042] S208: Determine the total loss value based on the number of training steps, the equalization loss value, and the main loss value.

[0043] In this step, a preset training step threshold is read, and the relationship between the current training step count and the threshold is determined. If the current training step count is less than the threshold, the system is in the warm-up phase. During the warm-up phase, the total loss value is only the primary loss value (prioritizing the convergence of the primary task). If the system is outside the warm-up phase (step count ≥ training step threshold), the primary loss value and the load balancing loss value are weighted and summed using preset weighting coefficients to obtain the total loss value, thus achieving collaborative optimization between the primary task and the load balancing task.

[0044] The training step threshold can be a preset value or a preset percentage of the maximum training step count. The total loss can be calculated as: Total Loss = Main Loss × ω1 + Equilibrium Loss × ω2, where ω1 and ω2 are preset weights, and their sum can be set to 1.

[0045] S209: Optimize the target expert module and routing module using total loss value.

[0046] In this step, based on the total loss value, the gradients (main loss gradient and equilibrium loss gradient; only the main loss gradient during the warm-up phase) of all learnable parameters of the target expert module and routing module are calculated using the backpropagation algorithm. An optimizer is then used to update the parameters.

[0047] As can be seen from the description of the above embodiments, the embodiments of this disclosure, through the collaborative optimization of "main loss + equalization loss", enable the routing module of the trained hybrid expert model to have the ability to dynamically equalize word allocation, which can not only ensure the accuracy of the output (main task convergence) but also avoid overload or idleness of the expert module. At the same time, by setting the termination condition of iterative training, the balance between model training efficiency and effect is ensured. This solves the problem of some experts being overloaded and some experts being idle due to the uneven word allocation in the training of traditional hybrid expert models (MoE), as well as the problem that single main loss training cannot take into account load balancing, which leads to increased communication costs and extended response cycles in multi-computing node scenarios. This improves the utilization rate of computing resources and increases the operating efficiency of system nodes.

[0048] In one possible implementation, after step S209 above, the method further includes: increasing the number of training steps by a preset value, and repeatedly executing the steps of obtaining training data and training steps until the steps of optimizing the target expert module and routing module using the total loss value are performed, until the number of training steps is greater than or equal to the training step threshold or the total loss value is less than the preset loss value threshold, and determining the hybrid expert model as the target hybrid expert model.

[0049] The process involves adding a preset value (e.g., 1, representing the completion of one iteration) to the current training step count. The "data acquisition, word conversion, loss calculation, and parameter update" process is repeatedly executed through a loop structure. After each iteration, a termination condition is determined (step count ≥ step count threshold or total loss value < loss threshold). If the condition is met, training stops, and the current model to be trained is saved as the final target hybrid expert model. The steps for acquiring training data and training steps can be described as step S200 above, and the steps for optimizing the target expert module and routing module using the total loss value can be described as step S209. During parameter updates, the updated parameters = original parameters - learning rate × gradient.

[0050] In one possible implementation, step S208 above, determining the total loss value based on the number of training steps, the equalization loss value, and the main loss value, includes:

[0051] S2081: If the number of training steps is less than the training step threshold, then the main loss value is determined as the total loss value.

[0052] In this step, a conditional statement can be used to compare the current number of training steps with a preset threshold. When the number of steps is less than the threshold, the warm-up phase logic is triggered, and the main loss value calculated in S206 is directly assigned to the total loss value. At this time, the equilibrium loss value does not participate in the total loss calculation, ensuring that the model prioritizes the main task in the early stages of training and quickly establishes basic response capabilities.

[0053] S2082: If the number of training steps is greater than or equal to the training step threshold, then the main loss value and the equilibrium loss value are weighted and summed to obtain the total loss value.

[0054] In this step, when the number of training steps reaches or exceeds a threshold, preset main loss weights and equilibrium loss weights are loaded. The product of the main loss value and its weight, and the product of the equilibrium loss value and its weight, are calculated separately using multiplication. The two products are then added together to obtain the total loss value. The weights are preset experimentally to balance the magnitudes of the two losses and prevent one loss from dominating the model's optimization direction.

[0055] As can be seen from the description of the above embodiments, the embodiments of this disclosure dynamically allocate the priority of the main and auxiliary tasks through the collaborative calculation of the main gradient and the balanced gradient, and generate a target gradient that takes into account both accuracy and balance; the routing module is optimized based on the target gradient, so that the parameter update direction of the routing module responds to the main task response requirements and the load balancing requirements at the same time, thereby improving the overall performance of the model. This solves the problem that the gradients of the main task and the load balancing task may conflict in the traditional optimization method, causing the model optimization direction to deviate, as well as the problem that the routing module optimization only focuses on a single task and cannot adapt to multi-expert collaborative work scenarios.

[0056] In one possible implementation, step S209 above, which optimizes the target expert module and the routing module using the total loss value, includes:

[0057] S2091: If the number of training steps is greater than or equal to the training step threshold, then the equalization gradient is determined based on the equalization loss value and the parameters of the routing module.

[0058] In this step, once the number of training steps is greater than or equal to the training step threshold, the equalization gradient calculation is initiated. Using the backpropagation algorithm, starting with the equalization loss value, the partial derivatives of the equalization loss with respect to each learnable parameter of the routing module are calculated. All partial derivatives form the equalization gradient vector. The direction of this vector indicates the direction of adjusting the routing parameters to reduce the equalization loss, and its strength indicates the magnitude of the adjustment. This vector only applies to the routing module parameters.

[0059] S2092: Determine the principal gradient based on the principal loss value and the parameters of the routing module.

[0060] In this step, the backpropagation algorithm is used, starting with the main loss value. Other parameters, such as the expert module, are fixed, and only the partial derivatives of the main loss with respect to the learnable parameters (weights, biases) of the routing module are calculated. An automatic differentiation framework is used to iterate through all parameters of the routing module, calculating the partial derivative value for each parameter. These values ​​are then organized into a gradient tensor along the parameter dimensions to obtain the principal gradient. This gradient reflects the direction and strength of adjusting the routing parameters to reduce the main loss.

[0061] S2093: Determine the norm ratio and phase based on the equilibrium gradient and principal gradient.

[0062] In this step, the Euclidean norms of the principal gradient and the equilibrium gradient are smoothed by exponential moving average, respectively, to obtain the smoothed principal gradient norm and the equilibrium gradient norm. The ratio of the two is the norm ratio. The dot product of the principal gradient and the equilibrium gradient is calculated and divided by the product of their Euclidean norms to obtain the phase (cosθ), which ranges from -1 to 1, reflecting the consistency of the gradient direction.

[0063] The norm ratio can be calculated as follows:

[0064]

[0065] In the formula, r represents the norm ratio, and EMA represents the exponential moving average smoothing process. Represents the principal gradient. This represents the equilibrium gradient.

[0066] The phase can be calculated as follows:

[0067]

[0068] In the formula, Represented as phase, Represents the principal gradient. This represents the equilibrium gradient.

[0069] S2094: Priority weight of primary task and priority weight of secondary task based on norm ratio and phase.

[0070] In this step, the priority weight of the main task is calculated based on the norm ratio and phase using a preset formula. Specifically, the difference between the norm ratio and the threshold is corrected by a sensitivity coefficient, then fused with the phase influence coefficient through a sigmoid mapping, and finally cropped to the 0-1 range. The priority weight of the auxiliary task is obtained by subtracting the priority weight of the main task from a preset value (usually 1), thus achieving dynamic allocation of weights.

[0071] S2095: Determine the target gradient based on phase, norm ratio, principal gradient, equilibrium gradient, primary task priority weight, and secondary task priority weight.

[0072] In this step, we first determine whether the phase and norm ratio meet the conflict condition (phase < phase threshold and norm ratio > norm ratio threshold). If they do, we calculate the projected value of the equilibrium gradient to remove components that conflict with the principal gradient. If the principal gradient norm is close to 0 (less than the divisor threshold), we directly use the principal gradient as the projected value. Then, we multiply the principal gradient by the primary task priority weight and the balanced gradient projection value by the secondary task priority weight, respectively, to obtain two weighted gradients. Finally, we add the two weighted gradients to obtain the target gradient, ensuring that the gradient update balances the accuracy of the primary task and load balancing.

[0073] S2096: Use the target gradient optimization routing module until the maximum number of optimization steps is reached or the main loss value is less than the preset main loss value threshold. Then, determine the hybrid expert model as the target hybrid expert model. Otherwise, increase the number of training steps by the preset value and repeat the steps of converting the input samples into tokens until the target gradient optimization routing module is used.

[0074] In this step, we first determine whether the phase and norm ratio meet the conflict condition (phase < phase threshold and norm ratio > norm ratio threshold). If they do, we calculate the projected value of the equilibrium gradient (removing components that conflict with the principal gradient). If the principal gradient norm is close to 0 (less than the divisor threshold), we directly use the principal gradient as the projected value. Then, we multiply the principal gradient by the priority weight of the main task and the projected value of the equilibrium gradient by the priority weight of the auxiliary task, respectively, to obtain two weighted gradients. Finally, we add the two weighted gradients to obtain the target gradient, ensuring that the gradient update takes into account both the accuracy of the main task and load balancing.

[0075] As can be seen from the description of the above embodiments, the embodiments of this disclosure accurately eliminate components in the equilibrium gradient that conflict with the main gradient by judging the conflict between phase and norm ratio, thus avoiding interference of conflicting gradients with model optimization; by weighted fusion of the main and auxiliary task weights, the target gradient can both ensure the improvement of the accuracy of the main task response and effectively optimize the expert load balancing, thereby improving the stability and efficiency of model training. This solves the problem that gradient updates may lead to a decrease in the accuracy of the main task when the main gradient and the equilibrium gradient conflict in direction, and also solves the problem that the projection calculation may be numerically unstable when the main gradient is too small.

[0076] In one possible implementation, step S2095 above, determining the target gradient based on phase, norm ratio, principal gradient, equalization gradient, primary task priority weight, and secondary task priority weight, includes:

[0077] S951: If the phase is less than the preset phase threshold and the norm ratio is greater than the preset norm ratio threshold, then the equalization gradient projection value is determined based on the principal gradient and the equalization gradient.

[0078] In this step, when the phase is less than a threshold (indicating severe gradient direction conflict) and the norm ratio is greater than a threshold (indicating excessively strong equilibrium gradient), the conflict resolution process is initiated. The projection coefficients are obtained by calculating the dot product of the principal gradient and the equilibrium gradient, and the square of the principal gradient norm. The projection coefficients are then rounded down to zero to obtain the target projection coefficients, retaining only the negative coefficients related to the conflict. The conflict correction is obtained by multiplying the target projection coefficients by the principal gradient. Finally, the conflict correction is subtracted from the equilibrium gradient to obtain the equilibrium gradient projection value after removing conflict components.

[0079] S952: If the principal gradient is less than the preset divisor threshold, then the principal gradient is determined as the projection value of the equilibrium gradient.

[0080] In this step, when the phase is less than a threshold (indicating gradient direction conflict) and the norm ratio is greater than a threshold (indicating a strong equilibrium gradient), the conflict resolution process is initiated. The projection coefficients are obtained by calculating the dot product of the principal gradient and the equilibrium gradient, and the square of the principal gradient norm. The projection coefficients are then rounded down to zero to obtain the target projection coefficients, retaining only the negative coefficients related to the conflict. The conflict correction is obtained by multiplying the target projection coefficients by the principal gradient. Finally, the conflict correction is subtracted from the equilibrium gradient to obtain the equilibrium gradient projection value after removing conflict components.

[0081] S953: Determine the target gradient based on the principal gradient, the projection values ​​of the equalization gradient, the priority weight of the primary task, and the priority weight of the secondary task.

[0082] In this step, element-wise multiplication is used to multiply the primary gradient with the primary task priority weight and the balanced gradient projection value with the secondary task priority, respectively, to obtain the primary task weighted gradient and the secondary task weighted gradient. Then, element-wise addition is used to add the two weighted gradients to obtain the target gradient. This gradient integrates the optimization requirements of the primary task and the load balancing task, guiding the routing module parameters to update in a collaborative optimization direction.

[0083] As described in the above embodiments, this disclosure uses weighted multiplication and addition operations to precisely fuse the optimization requirements of the main task and auxiliary task into a unified target gradient. This ensures that the weights of the main and auxiliary tasks can effectively regulate the contribution of the corresponding gradients, making the parameter update direction of the routing module more aligned with the dual objectives of "accuracy + balance". This solves the problem of ambiguous optimization objectives caused by the direct superposition of gradients from the main and auxiliary tasks, and also addresses the issue of weights failing to effectively contribute to gradient fusion, leading to ineffective optimization of load balancing or the accuracy of the main task.

[0084] In one possible implementation, step S953 above, determining the target gradient based on the principal gradient, the equalization gradient projection value, the primary task priority weight, and the secondary task priority weight, includes:

[0085] S9531: Multiply the primary task priority weights by the primary gradient to obtain the primary task weighted gradient.

[0086] In this step, the primary task priority weights are multiplied one by one with each element of the primary gradient to obtain the primary task weighted gradient. The larger the weight value, the stronger the contribution of the primary gradient to the final target gradient, ensuring that the primary task priority weights can effectively regulate the influence of the primary gradient.

[0087] S9532: Multiply the equalization gradient projection value by the secondary task priority to obtain the secondary task weighted gradient.

[0088] In this step, the priority of the auxiliary task is multiplied one by one with each element of the balanced gradient projection value to obtain the weighted gradient of the auxiliary task. The contribution intensity of the balanced gradient projection value is adjusted by the magnitude of the priority of the auxiliary task, so as to realize the reasonable influence of the load balancing task on the parameter update.

[0089] S9533: Add the weighted gradient of the main task to the weighted gradient of the auxiliary task to obtain the target gradient.

[0090] In this step, vector addition is used to add corresponding elements of the weighted gradients of the main task and the auxiliary task one by one to obtain the target gradient vector. This operation merges the optimization requirements of the two tasks into a unified gradient direction, ensuring that the routing module parameters respond simultaneously to the needs of improving the accuracy of the main task and optimizing load balancing when updated.

[0091] As can be seen from the description of the above embodiments, the embodiments of this disclosure achieve precise integration of the optimization requirements of the main task (output accuracy) and the auxiliary task (load balancing) by weighting the main task priority weight with the main gradient and the auxiliary task priority with the balanced gradient projection value respectively, and then fusing the two types of weighted gradients to obtain the target gradient. The contribution of the two types of tasks to the parameter update of the routing module can be flexibly adjusted by weight, ensuring that the parameter update direction takes into account both accuracy improvement and load balancing, and providing a reasonable gradient basis for the optimization of the routing module.

[0092] In one possible implementation, step S951 above, determining the projection value of the equilibrium gradient based on the principal gradient and the equilibrium gradient, includes:

[0093] S9511: The dot product of the principal gradient and the equilibrium gradient is determined as the target numerator value.

[0094] In this step, the vector dot product operation is used to multiply corresponding elements of the principal gradient vector and the equilibrium gradient vector one by one, and then sum all the product results to obtain a scalar value (target numerator value). The sign of this value reflects the directional relationship between the two gradients (positive value is cooperative, negative value is conflicting), and the magnitude reflects the degree of directional overlap.

[0095] S9512: Determine the target denominator value by the square of the Euclidean norm of the principal gradient.

[0096] In this step, the Euclidean norm of the principal gradient vector is first calculated. Specifically, all elements in the vector are squared, summed, and the square root is taken to obtain the norm value. Then, the obtained norm value is squared to obtain the target denominator value (equivalent to directly calculating the sum of squares of the principal gradient elements, simplifying the calculation process), which is used for the subsequent calculation of the projection coefficients.

[0097] S9513: The projection coefficient is obtained by dividing the target numerator value by the target denominator value.

[0098] In this step, the target numerator value (dot product result) is divided by the target denominator value (squared principal gradient norm) to obtain the scalar projection coefficient. This coefficient quantifies the projection intensity of the equilibrium gradient in the principal gradient direction, reflecting the degree of overlap between the equilibrium gradient and the principal gradient direction, and providing a basis for subsequent conflict identification.

[0099] S9514: The smaller value between the projection coefficient and the preset projection coefficient is determined as the target projection coefficient.

[0100] In this step, a preset projection coefficient (usually 0) is used. The smaller value between the "current projection coefficient" and the "preset projection coefficient" is selected as the target projection coefficient through a comparison operation. When the projection coefficient is negative (gradient conflict), the negative coefficient is retained. When the projection coefficient is non-negative, it is set to 0, thus implementing the filtering logic of retaining only conflicting components.

[0101] S9515: Multiply the target projection coefficients by the principal gradient to obtain the conflict correction amount.

[0102] In this step, scalar-vector multiplication is used to multiply the target projection coefficients (scalars) one by one with each element of the principal gradient vector to obtain the conflict correction vector. The direction of this vector is consistent with the principal gradient, and its magnitude is determined by the target projection coefficients, precisely corresponding to the components in the equilibrium gradient that conflict with the principal gradient.

[0103] S9516: Subtract the conflict correction amount from the equilibrium gradient to obtain the equilibrium gradient projection value.

[0104] In this step, vector subtraction is used to subtract the corresponding elements of the equilibrium gradient vector from the conflict correction vector one by one. This operation can eliminate the components in the equilibrium gradient that conflict with the main gradient, retaining only the cooperative or orthogonal components, and obtaining the equilibrium gradient projection value, thus avoiding conflicts between load balancing optimization and main task optimization.

[0105] Steps S9511 to S9516 above can be expressed as follows:

[0106]

[0107] In the formula, Indicates the equalization gradient projection value. Represents the principal gradient. This represents the equilibrium gradient.

[0108] As described in the above embodiments, this disclosure improves the effectiveness of gradient updates by accurately filtering out and removing components in the equilibrium gradient that conflict with the main gradient, while retaining cooperative or orthogonal components. This ensures that the optimized equilibrium gradient no longer interferes with the main task optimization, while still maintaining the load balancing control function. This solves the problem that load balancing optimization weakens the convergence effect of the main task when the equilibrium gradient and the main gradient conflict.

[0109] In one possible implementation, step S2094 above, determining the priority weight of the primary task and the priority weight of the secondary task based on the norm ratio and phase, includes:

[0110] S941: Determine the priority weight of the primary task based on the norm ratio and phase.

[0111] In this step, the difference between the norm ratio and the preset norm ratio threshold is first calculated. This difference is then multiplied by the sensitivity coefficient to obtain the corrected difference. The corrected difference is mapped to the 0-1 interval using the sigmoid function, yielding the mapped norm ratio value. The product of the mapped norm ratio value and (1 - phase influence coefficient) is calculated, and then the phase influence coefficient is added to obtain the weights to be clipped. Finally, the clip function is used to restrict the weights to be clipped to the 0-1 interval, resulting in the primary task priority weights, thus achieving coordinated control of the weights by the norm ratio and phase.

[0112] The priority weight of the primary task can be calculated as follows:

[0113]

[0114] In the formula, Indicates the priority weight of the main task. This represents the clip function, where r represents the norm ratio. This represents the preset norm to threshold, k represents the sensitivity coefficient, and σ represents the sigmoid function. This represents the phase influence coefficient.

[0115] S942: The difference between the preset value and the priority weight of the main task is determined as the priority weight of the auxiliary task.

[0116] In this step, a fixed scalar value (which can be 1) is preset. The priority weight of the main task is subtracted from this preset value by scalar subtraction. The result is the priority weight of the secondary task.

[0117] As can be seen from the description of the above embodiments, the embodiments of this disclosure, through norm ratio difference correction, sigmoid smoothing mapping and clipping, ensure that the priority weight of the main task is always in the legal range of 0-1 and can smoothly respond to changes in norm ratio; by integrating the phase influence coefficient, the weights take into account both gradient strength and direction relationship, thereby improving the rationality and accuracy of weight allocation.

[0118] In one possible implementation, step S2093 above, determining the norm ratio and phase based on the equilibrium gradient and the principal gradient, includes:

[0119] S9311: Determine the smoothed value of the principal gradient and obtain the smoothed value of the principal gradient.

[0120] In this step, the exponential moving average (EMA) algorithm is used to smooth and filter the random noise of the single-round gradient norm, so that the smoothed value can better reflect the long-term strength trend of the principal gradient.

[0121] S9312: Determine the smoothing value of the equilibrium gradient to obtain the smoothing value of the equilibrium gradient.

[0122] This step is similar to step S9311 above, and will not be repeated here.

[0123] S9313: The ratio of the principal gradient smoothing value to the equilibrium gradient smoothing value is determined as the norm ratio.

[0124] In this step, a scalar division operation is performed, dividing the smoothed value of the equilibrium gradient by the smoothed value of the principal gradient to obtain the norm ratio. This ratio quantifies the relative strength of the equilibrium gradient and the principal gradient. If the ratio is greater than 1, it indicates that the equilibrium gradient is stronger than the principal gradient; conversely, if the ratio is less than 1, the principal gradient is stronger, providing a strength basis for weight allocation.

[0125] S9314: Determine the dot product of the principal gradient and the equilibrium gradient as the dot product value.

[0126] In this step, a vector dot product operation is performed, which involves multiplying the corresponding elements of the principal gradient and the equilibrium gradient and then summing them to obtain the dot product value.

[0127] S9315: The dot product of the Euclidean norm of the principal gradient and the Euclidean norm of the equilibrium gradient is defined as the norm dot product.

[0128] In this step, the Euclidean norms of the principal gradient and the equilibrium gradient are calculated separately, and then the two norm values ​​are multiplied by a scalar to obtain the norm dot product. This value is used as the denominator for the phase calculation to normalize the dot product value, so that the phase result is limited to the interval [-1, 1].

[0129] S9316: The phase is obtained by dividing the dot product by the norm dot product.

[0130] In this step, a scalar division operation is performed, dividing the dot product by the norm dot product to obtain the phase (cosθ). This result is the cosine of the angle between the two gradient directions, ranging from [-1, 1]. The closer to 1, the more compatible the directions; the closer to -1, the more conflicting the directions; and the closer to 0, the more orthogonal the directions.

[0131] As can be seen from the description of the above embodiments, the embodiments of this disclosure filter out random noise in the norms of the principal gradient and the equilibrium gradient through smoothing processing, so that the norm ratio can stably reflect the long-term strength relationship between the two; at the same time, the consistency of gradient direction is accurately quantified by the operation of dot product and norm product, providing a basis with both stability and reliability for the dynamic calculation of the priority weight of the subsequent main task, thereby improving the stability and convergence speed of model training.

[0132] In one possible implementation, step S932 above, determining the priority weight of the primary task based on the norm ratio and phase, includes:

[0133] S9321: The norm ratio difference is obtained by subtracting the preset norm ratio threshold from the norm ratio.

[0134] In this step, a scalar subtraction operation is performed, subtracting a preset norm ratio threshold (e.g., 0.5, 0.45, etc., which serves as a benchmark for judging whether the equilibrium gradient is too strong) from the currently calculated norm ratio. The resulting norm ratio difference reflects the degree of deviation of the current norm ratio from the norm ratio threshold; a positive value indicates that the equilibrium gradient strength exceeds the norm ratio threshold, while a negative value indicates that it is below the norm ratio threshold.

[0135] S9322: Multiply the norm ratio difference by the preset sensitivity coefficient to obtain the corrected norm ratio difference.

[0136] In this step, a scalar multiplication operation is performed, multiplying the norm ratio difference by a preset sensitivity coefficient. By amplifying or reducing the influence of the norm ratio difference through the coefficient, changes in the norm ratio become more sensitive to subsequent weight calculations (the larger the coefficient, the higher the sensitivity), adapting to the optimization needs of different scenarios.

[0137] S9323: Map the modified norm ratio difference to a preset numerical range to obtain the norm ratio mapping value.

[0138] In this step, the sigmoid activation function is used to map the modified norm ratio difference to the interval [0, 1]. The mapped norm ratio value achieves a smooth transition, avoiding sudden changes in weights due to small fluctuations in the norm ratio.

[0139] S9324: Multiply the norm ratio mapping value by the target difference, and then add it to the preset phase influence coefficient to obtain the pruning weight, where the target difference is the difference between the target preset value and the phase influence coefficient.

[0140] In this step, the target difference is the difference between the "target preset value" and the "phase influence coefficient" (the target preset value can be 1). The entire calculation process strictly follows the step definition. By performing a scalar multiplication operation between the norm ratio mapping value and the target difference, and then adding the intermediate product result with the preset phase influence coefficient, the final pruning weight is obtained.

[0141] S9325: Trim the weights to be trimmed to obtain the priority weights of the main task.

[0142] In this step, the `clip` function is used to limit the range of the weights to be clipped, with a preset clipping interval of [0,1]. If the weight to be clipped is less than 0, 0 is output. If it is greater than 1, 1 is output. If it is within the 0-1 interval, the original weight value is output directly. Clipping ensures that the priority weights of the main task are valid probability values, avoiding extreme values ​​that could lead to abnormal gradient updates.

[0143] As can be seen from the description of the above embodiments, the embodiments of this disclosure amplify the influence of gradient strength differences through norm ratio difference calculation and sensitivity correction, achieve smooth weight transition and avoid abrupt changes through sigmoid mapping, then fuse phase influence coefficient to take into account gradient direction relationship, and finally ensure the legality of weights through pruning, forming a complete dynamic calculation process for priority weights of the main task. The final weights can accurately adapt to the dynamic changes of gradients, providing a reliable basis for subsequent gradient fusion of main and auxiliary tasks, effectively improving the rationality of weight allocation and the stability of model training, ensuring the accuracy of the main task while taking into account load balancing optimization.

[0144] In one possible implementation, step S2092 above, determining the principal gradient based on the principal loss value and the parameters of the routing module, includes:

[0145] S921: The principal gradient is obtained by taking the partial derivative of the principal loss value with respect to the parameters of the routing module.

[0146] In this step, the principal loss is used as the objective function, and the parameters of the routing module are used as the variables for differentiation. The first-order partial derivatives of the objective function with respect to each variable are calculated. For example, for the weight matrix W of the fully connected layer of the routing module, the partial derivatives of the principal loss with respect to each element of W are calculated. All partial derivatives form the principal gradient matrix with the same dimension as W, and are finally summarized into a complete principal gradient tensor.

[0147] As can be seen from the description of the above embodiments, the embodiments of this disclosure accurately calculate the partial derivative of the main loss with respect to the routing module parameters through an automatic differentiation mechanism. The obtained main gradient can accurately lock the direction and intensity of adjusting the routing parameters to reduce the main loss. This provides a reliable basis for the main task gradient for subsequent calculation of the main task priority weight and fusion of the main and auxiliary task gradients, effectively ensuring the accurate convergence of the main task and avoiding insufficient accuracy of the main task due to deviations in the calculation of the main gradient.

[0148] In one possible implementation, in step S2091 above, if the number of training steps is greater than or equal to a training step threshold, then the equilibrium gradient is determined based on the equilibrium loss value and the parameters of the routing module, including:

[0149] S911: If the number of training steps is greater than or equal to the training step threshold, the partial derivative of the equalization loss value with respect to the parameters of the routing module is used to obtain the equalization gradient.

[0150] In this step, using an automatic differentiation framework, the parameters of other modules are fixed, and only the partial derivatives of the equilibrium loss value are calculated with respect to the parameters of the routing module. The weight matrix and bias vector of the fully connected layer of the routing module are traversed, and the partial derivative value of each element is calculated. All partial derivatives are organized into an equilibrium gradient tensor according to the parameter dimensions to obtain the final equilibrium gradient.

[0151] As can be seen from the description of the above embodiments, the embodiments of this disclosure calculate the partial derivative of the main loss with respect to the routing module parameters by differentiation. The obtained main gradient can accurately lock the direction and intensity of adjusting the routing parameters to reduce the main loss. This provides a reliable basis for the main task gradient for subsequent calculation of the main task priority weight and fusion of the main and auxiliary task gradients, effectively ensuring the accurate convergence of the main task and avoiding insufficient accuracy of the main task due to deviation in the calculation of the main gradient.

[0152] In one possible implementation, step S204 above, determining the target expert module based on the expert assignment probability, includes:

[0153] S2041: Identify at least one expert module with the highest expert assignment probability as the target expert module.

[0154] In this step, for a single word, all expert assignment probabilities corresponding to it are traversed, and one or more expert modules with the highest probability values ​​are selected (the specific number is determined by the preset hyperparameter k). Through index matching of expert modules, these expert modules with the highest probabilities are directly marked as target expert modules, ensuring that the word is assigned to the expert module with the highest suitability for processing.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0156] In one possible implementation, after determining the hybrid expert model as the target hybrid expert model in step S209 above, the method further includes:

[0157] S221: Receive consultation statements sent by the consultation terminal.

[0158] In this step, a preset client-server communication protocol (such as HTTP or TCP) is used. The server starts a port listening mechanism. When the client initiates a request, the server receives the text data query statement transmitted by the client through the specified port.

[0159] S222: Convert the query statement into input tokens.

[0160] In this step, the pre-trained word segmenter consistent with the model training phase is invoked to preprocess the consultation statement. Then, the preprocessed text is split into the smallest semantic units (sub-words / words). Finally, through the word segmenter's vocabulary mapping, each semantic unit is converted into a word unit that the model can recognize. At the same time, special words units required by the model (such as start words, separator words, etc.) are added to generate a word unit sequence that conforms to the model input format, thus obtaining the word unit to be input.

[0161] Preprocessing methods may include cleaning special characters, removing extra spaces, and segmenting sentences.

[0162] S223: Input the word to be input into the target hybrid expert model to obtain the consultation result.

[0163] In this step, after the input word sequence is fed into the model, it is first converted into fixed-dimensional embedding vectors by the embedding layer. The embedding vectors are then passed to the routing module, which selects a load-balanced expert module. The word vectors are assigned to the selected expert module, where feature extraction and semantic understanding are performed by the expert module's network layers (such as Transformer layers and fully connected layers). The features output by the expert module are then aggregated by a fusion layer (weighted summation according to the assigned probabilities), and the input-output layer generates a word sequence for the predicted text. Finally, the word sequence is converted into natural language text (consultation results) through the inverse mapping of the word segmenter. Because the model is trained with load balancing, the routing module can dynamically and evenly distribute words, avoiding expert overload or idleness.

[0164] The routing module can calculate the allocation probability of each expert module using a fully connected layer combined with an activation function. The output layer can include a fully connected layer and a normalized exponential function layer.

[0165] S224: Send the consultation results to the consultation terminal so that the consultation terminal can output the consultation results.

[0166] In this step, the server encapsulates the consultation result into a response data packet according to a preset communication protocol and sends it to the designated port of the consulting terminal through the previously established communication link. After receiving the response data packet, the terminal parses the text content in the data packet and outputs the consultation result through the terminal's display interface, completing the interaction loop.

[0167] As can be seen from the description of the above embodiments, the present disclosure embodiments use a target hybrid expert model trained by a load balancing method to respond to consultation information, which can not only ensure the accuracy of consultation response (main task convergence), but also avoid expert module overload or idleness, so that the model can also ensure high node computing efficiency and computing power utilization during use.

[0168] Figure 3 This is a schematic diagram of the load balancing training device provided in this application. Figure 3 As shown, embodiments of this application also provide a load balancing training device 300, which includes: a data acquisition module 301, a sample conversion module 302, a probability acquisition module 303, a load determination module 304, a module determination module 305, an information acquisition module 306, a main loss determination module 307, a balance loss determination module 308, a total loss determination module 309, and a model correction module 310.

[0169] The data acquisition module 301 is used to acquire training data and training steps, wherein the training data includes input samples and output samples.

[0170] The sample conversion module 302 is used to convert the input sample into a word.

[0171] The probability acquisition module 303 is used to input the word into the routing module of the hybrid expert model and obtain the expert assignment probability corresponding to the expert module output by the routing module.

[0172] The load determination module 304 is used to determine the load corresponding to the expert module based on the expert allocation probability.

[0173] Module 305 is used to determine the target expert module based on the expert allocation probability.

[0174] The information acquisition module 306 is used to input lexical units into the target expert module and obtain the target output result output by the target expert module.

[0175] The main loss determination module 307 is used to determine the main loss value based on the target output result and the output sample.

[0176] The balancing loss determination module 308 is used to determine the balancing loss value based on the load corresponding to the expert module.

[0177] The total loss determination module 309 is used to determine the total loss value based on the number of training steps, the equalization loss value, and the main loss value.

[0178] The model correction module 310 is used to optimize the target expert module and the routing module using the total loss value.

[0179] In one possible implementation, the main loss determination module 307 is used to determine the main loss value as the total loss value if the number of training steps is less than the training step threshold; and to obtain the total loss value by weighted summation of the main loss value and the equilibrium loss value if the number of training steps is greater than or equal to the training step threshold.

[0180] In one possible implementation, the model correction module 310 is used to determine the equilibrium gradient based on the equilibrium loss value and the parameters of the routing module if the number of training steps is greater than or equal to a training step threshold; determine the main gradient based on the main loss value and the parameters of the routing module; determine the norm ratio and phase based on the equilibrium gradient and the main gradient; determine the main task priority weight and the auxiliary task priority based on the norm ratio and the phase; determine the target gradient based on the phase, norm ratio, main gradient, equilibrium gradient, main task priority weight and auxiliary task priority; and optimize the routing module using the target gradient until the maximum number of optimization steps is reached or the main loss value is less than a preset main loss value threshold, at which point the hybrid expert model is determined as the target hybrid expert model; otherwise, the number of training steps is increased by a preset value, and the steps of converting input samples into tokens and then optimizing the routing module using the target gradient are repeated.

[0181] In one possible implementation, the model correction module 310 is used to determine the equilibrium gradient projection value based on the principal gradient and the equilibrium gradient if the phase is less than a preset phase threshold and the norm ratio is greater than a preset norm ratio threshold; if the principal gradient is less than a preset divisor threshold, the principal gradient is determined as the equilibrium gradient projection value; and the target gradient is determined based on the principal gradient, the equilibrium gradient projection value, the primary task priority weight, and the secondary task priority weight.

[0182] In one possible implementation, the model correction module 310 is used to multiply the main task priority weights with the main gradient to obtain the main task weighted gradient; multiply the equalization gradient projection value with the secondary task priority to obtain the secondary task weighted gradient; and add the main task weighted gradient and the secondary task weighted gradient to obtain the target gradient.

[0183] In one possible implementation, the model correction module 310 is used to determine the target numerator value by the dot product of the principal gradient and the equilibrium gradient; determine the target denominator value by the square of the Euclidean norm of the principal gradient; obtain the projection coefficient by dividing the target numerator value by the target denominator value; determine the smaller value between the projection coefficient and the preset projection coefficient as the target projection coefficient; multiply the target projection coefficient by the principal gradient to obtain the conflict correction amount; and subtract the conflict correction amount from the equilibrium gradient to obtain the equilibrium gradient projection value.

[0184] In one possible implementation, the model correction module 310 is used to determine the priority weight of the main task based on the norm ratio and phase; and to determine the difference between the preset value and the priority weight of the main task as the priority weight of the auxiliary task.

[0185] In one possible implementation, the model correction module 310 is used to determine the smoothed value of the principal gradient, obtain the smoothed value of the principal gradient; determine the smoothed value of the equilibrium gradient, obtain the smoothed value of the equilibrium gradient; determine the ratio of the smoothed value of the principal gradient to the smoothed value of the equilibrium gradient as the norm ratio; determine the dot product of the principal gradient and the equilibrium gradient as the dot product value; determine the dot product of the Euclidean norm of the principal gradient and the Euclidean norm of the equilibrium gradient as the norm dot product; and obtain the phase by dividing the dot product value by the norm dot product.

[0186] In one possible implementation, the model correction module 310 is used to obtain a norm ratio difference by subtracting a preset norm ratio threshold from the norm ratio; multiply the norm ratio difference by a preset sensitivity coefficient to obtain a corrected norm ratio difference; map the corrected norm ratio difference to a preset numerical range to obtain a norm ratio mapping value; multiply the norm ratio mapping value by a target difference and add it to a preset phase influence coefficient to obtain the weight to be pruned, wherein the target difference is the difference between the target preset value and the phase influence coefficient; and prune the weight to be pruned to obtain the main task priority weight.

[0187] In one possible implementation, the model correction module 310 is used to take the partial derivative of the main loss value with respect to the parameters of the routing module to obtain the main gradient.

[0188] In one possible implementation, the model correction module 310 is used to obtain the equilibrium gradient by taking the partial derivative of the equilibrium loss value with respect to the parameters of the routing module if the number of training steps is greater than or equal to the training step threshold.

[0189] For a description of the features in the embodiment corresponding to the consultation response device, please refer to the relevant description of the embodiment corresponding to the consultation response method, which will not be repeated here.

[0190] In one possible implementation, the load balancing training device 300 further includes a consultation processing module 311.

[0191] The consultation processing module 311 receives consultation statements sent by the consultation terminal. It converts the consultation statements into input tokens. These tokens are then input into the consultation response model, which is trained using a load-balanced training process, to obtain the consultation result. Finally, the consultation result is sent to the consultation terminal so that the terminal can output the consultation result.

[0192] For a description of the features in the embodiment corresponding to the consultation response device, please refer to the relevant description of the embodiment corresponding to the consultation response method, which will not be repeated here.

[0193] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Figure 4As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus.

[0194] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the above-described consultation and response method embodiment.

[0195] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0196] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0197] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0198] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0199] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the load balancing training method for hybrid expert models at runtime.

[0200] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0201] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described consultation and response method embodiments.

[0202] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above-described consultation and response method embodiments.

[0203] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0204] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0205] The above provides a detailed description of the load balancing training method and device for a hybrid expert model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A load-balanced training method for a hybrid expert model, characterized in that, include: Acquire training data and training steps, wherein the training data includes input samples and output samples; Convert the input sample into word units; The word is input into the routing module of the hybrid expert model to obtain the expert assignment probability corresponding to the expert module output by the routing module. The load corresponding to the expert module is determined based on the expert allocation probability. The target expert module is determined based on the expert allocation probability. The word units are input into the target expert module to obtain the target output result output by the target expert module; Based on the target output and the output sample, determine the main loss value; The balancing loss value is determined based on the load corresponding to the expert module; The total loss value is determined based on the number of training steps, the equilibrium loss value, and the main loss value. The total loss value is used to optimize the target expert module and the routing module.

2. The method according to claim 1, characterized in that, The step of determining the total loss value based on the number of training steps, the equilibrium loss value, and the main loss value includes: If the number of training steps is less than the training step threshold, then the main loss value is determined as the total loss value; If the number of training steps is greater than or equal to the training step threshold, then the main loss value and the equilibrium loss value are weighted and summed to obtain the total loss value.

3. The method according to claim 2, characterized in that, The optimization of the target expert module and the routing module using the total loss value includes: If the number of training steps is greater than or equal to the training step threshold, then the equalization gradient is determined based on the equalization loss value and the parameters of the routing module. The main gradient is determined based on the main loss value and the parameters of the routing module; Determine the norm ratio and phase based on the equilibrium gradient and the principal gradient; Based on the norm ratio and the phase, the priority weight of the primary task and the priority weight of the secondary task are determined; The target gradient is determined based on the phase, the norm ratio, the principal gradient, the equilibrium gradient, the primary task priority weight, and the secondary task priority. The routing module is optimized using the target gradient until the maximum number of optimization steps is reached or the main loss value is less than a preset main loss value threshold. The hybrid expert model is then identified as the target hybrid expert model. Otherwise, the number of training steps is increased by a preset value, and the steps of converting the input samples into tokens and then optimizing the routing module using the target gradient are repeated.

4. The method according to claim 3, characterized in that, The step of determining the target gradient based on the phase, the norm ratio, the principal gradient, the equalization gradient, the primary task priority weight, and the secondary task priority weight includes: If the phase is less than a preset phase threshold and the norm ratio is greater than a preset norm ratio threshold, then the equalization gradient projection value is determined based on the principal gradient and the equalization gradient. If the principal gradient is less than a preset divisor threshold, then the principal gradient is determined as the projection value of the equilibrium gradient. The target gradient is determined based on the principal gradient, the projection value of the equalization gradient, the priority weight of the primary task, and the priority weight of the secondary task.

5. The method according to claim 4, characterized in that, The step of determining the target gradient based on the principal gradient, the projection value of the equalization gradient, the priority weight of the primary task, and the priority weight of the secondary task includes: Multiply the primary task priority weight by the primary gradient to obtain the primary task weighted gradient; Multiply the equalization gradient projection value by the secondary task priority to obtain the secondary task weighted gradient; The target gradient is obtained by adding the weighted gradient of the main task to the weighted gradient of the auxiliary task.

6. The method according to claim 4, characterized in that, Determining the equalization gradient projection value based on the principal gradient and the equalization gradient includes: The dot product of the principal gradient and the equilibrium gradient is determined as the target numerator value; The square of the Euclidean norm of the principal gradient is determined as the target denominator value; The projection coefficient is obtained by dividing the target numerator value by the target denominator value; The smaller value between the projection coefficient and the preset projection coefficient is determined as the target projection coefficient; Multiplying the target projection coefficients by the principal gradient yields the conflict correction amount; The equalization gradient projection value is obtained by subtracting the conflict correction amount from the equalization gradient.

7. The method according to claim 4, characterized in that, The step of determining the priority weight of the primary task and the priority weight of the secondary task based on the norm ratio and the phase includes: The priority weight of the main task is determined based on the norm ratio and the phase. The difference between the preset value and the priority weight of the main task is determined as the priority weight of the auxiliary task.

8. The method according to claim 3, characterized in that, The step of determining the norm ratio and phase based on the equalization gradient and the principal gradient includes: Determine the smoothed value of the principal gradient to obtain the smoothed principal gradient value; Determine the smoothed value of the equilibrium gradient to obtain the smoothed value of the equilibrium gradient; The ratio of the principal gradient smoothing value to the equilibrium gradient smoothing value is determined as the norm ratio; The dot product of the principal gradient and the equilibrium gradient is determined as the dot product value; The dot product of the Euclidean norm of the principal gradient and the Euclidean norm of the equilibrium gradient is defined as the norm dot product: The phase is obtained by dividing the dot product value by the norm dot product.

9. The method according to claim 7, characterized in that, The step of determining the priority weight of the primary task based on the norm ratio and the phase includes: The norm ratio difference is obtained by subtracting the preset norm ratio threshold from the norm ratio. The modified norm ratio difference is obtained by multiplying the norm ratio difference by a preset sensitivity coefficient. The modified norm ratio difference is mapped to a preset numerical range to obtain the norm ratio mapping value; Multiply the norm ratio mapping value by the target difference, and then add it to the preset phase influence coefficient to obtain the pruning weight, wherein the target difference is the difference between the target preset value and the phase influence coefficient; The weights to be clipped are used to obtain the priority weights of the main task.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the load balancing training method as described in any one of claims 1 to 9.

Images