A hybrid routing optimization method based on a MoE architecture large model

Abstract

The application relates to a mixed routing optimization method of a large language model MoE architecture, which fuses a local selection mechanism of a gate routing and a task self-adaptive strategy of a hierarchical variable k value, realizes dynamic identification and routing optimization of low-quality experts in an inference stage, and performs top-k expert selection in the inference stage. The application performs sorting analysis on candidate expert weights, calculates relative weight differences of adjacent experts, sets a threshold, and implements routing truncation when the differences are too large. The threshold setting is based on the premise that the accuracy of the current same type of task is not affected, so as to avoid unreasonable allocation of computing resources between low-weight experts and high-weight experts. At the model structure level, different Transformer layers have differential function representation and contribution characteristics, and accordingly, the k value of the MoE module is adaptively adjusted in layers. A multi-objective optimization method is introduced to define the hierarchical k value, and the inference accuracy and the computing cost are taken as optimization objectives to determine the optimal k value configuration of each layer. The application significantly improves the execution efficiency and resource utilization of the large language model MoE network.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, and specifically to a hybrid routing optimization method based on a MoE architecture large model. Background Technology

[0002] The advent of large language models represents a significant breakthrough in the fields of artificial intelligence and natural language processing. In recent years, large language models based on the Transformer architecture, supported by big data technologies, have gained access to a vast amount of specialized corpora, demonstrating significant advantages in long-text context processing and semantic understanding. As the demands on model capabilities and application scenarios continue to expand, the model's emergent capabilities and application effects are largely related to the model's parameter scale, and the continuous development of large language models places increasingly higher demands on computing resources and hardware. The MoE architecture for large language models achieves specialization in model task inference and saves computing resources, alleviating some of the challenges in the development of large language models.

[0003] However, the traditional MoE architecture still faces the problem of using the same top-k strategy for different tokens at different Transformer layers, which leads to the introduction of many unnecessary low-weight network calculations, increases the consumption of computing resources, and causes more uncertainty in the model's output.

[0004] Given this situation, dynamically defining the number of activated expert networks is clearly an effective strategy. Based on the current MoE architecture with fine-grained expert partitioning, the number of activated expert networks has increased from 1-2 to 4-8. More and smaller expert networks indicate that a more flexible strategy should be adopted in the dynamic activation process. That is, the dynamic selection of experts needs to be performed while ensuring that the model's capabilities are not significantly affected. Too few experts will lead to insufficient model capabilities, while too many experts will result in additional computational overhead. Therefore, determining where and how to define the number of activated experts becomes the main problem that needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a hybrid routing optimization method based on a large MoE architecture model, which reduces unnecessary computational losses and increases model efficiency by dynamically adjusting the number of expert networks selected by its routing gating in both local and global modes.

[0006] To achieve the above objectives, this invention provides a hybrid routing optimization method based on a large MoE architecture model, comprising the following steps:

[0007] Step 1: Extract the expert weights after passing through the gating matrix in the routing part of the MoE network;

[0008] Step 2: Determine whether there is a situation where the relative difference between the weights of adjacent experts is too large after sorting the first k expert networks. If so, cut off the subsequent experts.

[0009] Step 3: After obtaining the optimal value of local route pruning under the current task, the k value of hierarchical route selection is further defined by a multi-objective optimization algorithm;

[0010] Step 4: Apply the previously obtained local routing gating truncation threshold and the level k value used when not truncating in the specific large language model task inference to complete the inference optimization.

[0011] Optionally, the expert weights extracted in step 1 are vectors representing the weights of all expert networks obtained by passing the intermediate vector through the gate matrix of the MoE network under the current token batch in the decoder stage of the large model. The length of the vector is related to the batch size of the tokens and the preset number of experts in the selected model.

[0012] Optionally, in step 2, determining when the relative difference in weights between adjacent experts is too large involves calculating the relative proportion of the weight decrease from one expert to the next in the experts sorted in descending order. The truncation criterion is expressed as follows:

[0013]

[0014] in, This represents the current input in the nth MoE layer, where the weights of the selected top-k experts are arranged in descending order. Individual weighted activation values; specific thresholds The value will vary depending on the model and the task.

[0015] Optionally, the multi-objective optimization algorithm used in step 3 is the NSGA-II algorithm. The objective function is to maximize the task accuracy while minimizing the number of experts used by the model. The input is a one-dimensional vector whose length corresponds to the number of layers in the model. Each element represents the top-k value adopted by that layer.

[0016] Optionally, the multi-objective optimization algorithm is based on step 2 and is responsible for determining the optimal value of the route truncation threshold at the local level, and defining the global level k value in subsequent algorithm optimization.

[0017] Optionally, the multi-objective optimization process includes the following steps:

[0018] Step 3.1: First, the algorithm generates an initial population, inputs the round value into the evaluation model, and allows the model to implement the corresponding k value at each layer in the specific task, thereby obtaining the task accuracy and expert usage rate of the evaluation.

[0019] Step 3.2: Perform non-dominated sorting and crowding calculation on the objective function results of the initial population to generate the first generation parent population;

[0020] Step 3.3: From the current parent population, generate a child population of the same size by calculating the sorting level and crowding degree, crossover, and mutation genetic operators;

[0021] Step 3.4: Merge the parent and offspring populations to form a temporary population of size 2N. Perform fast non-dominated sorting, and then select N individuals based on sorting hierarchy and crowding to form the new generation of parent population.

[0022] Step 3.5: Use the newly generated parent population as the parent population for the next iteration, return to step 3.3, and continue until the termination condition is met, i.e., the maximum number of iterations is reached, or the solution set no longer improves significantly.

[0023] Optionally, in step 4, when using the optimized model, the Pareto front of the optimal cutoff threshold and level k value obtained by different models in different tasks will be applied specifically, rather than seeking the same value for all models and all tasks.

[0024] This invention provides a hybrid routing optimization method based on a large MoE architecture model. It combines local selection via gated routing with task-oriented adaptation of the k-value at each layer to dynamically identify low-quality expert networks during model execution. The basic idea is to perform a secondary screening of the selected k experts in the top-k routing strategy based on a weight decrease threshold. When the routing weight of the next expert among the k experts significantly decreases compared to the current expert's weight (i.e., the decrease exceeds a certain threshold), the importance of subsequent experts is greatly reduced, thus balancing computational cost with expert participation. This is based on the understanding that information entropy increases progressively in different layers of the MoE network—meaning the last layer has the least confidence in its predictions, while earlier layers tend to be more stable. For a specific task domain, the token will select a few experts with lower acceptance rates in the first few layers of the MoE network, while requiring more uncertainty in some later layers. This raises the question of whether the k-value for each layer is also dynamically adjusted based on its information entropy. This invention introduces a fast non-dominated sorting genetic algorithm II to explore the benefits of different levels of k values ​​to the accuracy and computational cost of experts, thereby minimizing the computational cost of the MoE model while maintaining accuracy. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart illustrating a hybrid routing optimization method based on a large MoE architecture model according to the present invention.

[0027] Figure 2 This is a schematic diagram comparing the activation weight values ​​of a token in two different expert layers in an embodiment of the present invention.

[0028] Figure 3 This is a comparative schematic diagram of the methods for dividing absolute thresholds and relative thresholds in embodiments of the present invention.

[0029] Figure 4 This is a schematic diagram of the workflow for determining and pruning the local routing threshold in this invention.

[0030] Figure 5 This is a diagram showing the results of local routing threshold pruning in specific experiments of this invention on different datasets of different models.

[0031] Figure 6 This is a flowchart illustrating the hierarchical k-value delimitation method for global multi-objective optimization in this invention.

[0032] Figure 7 This is a schematic diagram of the Pareto solution set thermodynamics for the hierarchical k-value delimitation of global multi-objective optimization in this invention.

[0033] Figure 8 This is a diagram showing the computational gains achieved by the final experimental results of this invention in the MoE network part. Detailed Implementation

[0034] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0035] Before proceeding, some background technical knowledge and terminology will be introduced:

[0036] The basic structure of the Mixture-of-Experts (MoE) architecture consists of a gated routing layer and several expert networks. The core idea is to select k experts from these networks to handle the current task, and then perform weighted processing. In past research, MoE has emerged as a promising architectural solution to address scaling challenges in machine learning. Initially developed as a method for learning subtasks in neural networks, many MoE-based models have been developed over the years. With the development of large language models, the MoE architecture has also been extensively used in the optimization of Transformer modules. The core principle is to distribute the model's capacity among multiple specialized subnetworks or experts and selectively activate relevant experts for each input through a learned gating mechanism. This method allows the model to maintain a large number of parameters while keeping computational costs manageable through sparse activation of some parameters. Recent implementations, such as Mixtral 8x7B, DeepSeekMoE, and Qwen3-30A-3B, have demonstrated the effectiveness of this strategy in scaling language models to trillions of parameters while maintaining reasonable computational requirements.

[0037] In the large-scale MoE architecture, each layer of the MoE network contains a routing gate G, which filters the expert networks to be selected at the current layer based on the current input token. After passing through this subset of expert networks, the weights of each expert given the input are calculated. Then execute all the results of the expert network. Aggregation is performed. Formally, the output of a MoE layer is:

[0038] .

[0039] The routing network G is a top-k routing network with parameters Wg and Wnoise. The result is the activation weight value of the expert selected by the gated routing. The activation weight value is a scalar value calculated by the routing network based on the input token and then normalized by Softmax. It represents the importance or selection probability of each expert to the current input. It can be expressed as:

[0040]

[0041] in, Representing the standardized MoE noise paradigm, TopK sets all elements except the K largest values ​​in the vector to zero, while Softplus is a smooth approximation of the ReLU function.

[0042] .

[0043] Please refer to the flowchart of the method described in this article. Figure 1This invention provides a hybrid routing optimization method based on a large MoE architecture model, comprising the following steps:

[0044] S1: Extract the expert weights after passing through the gated matrix in the routing part of the MoE network;

[0045] S2: Determine whether there is a situation where the relative difference in weights of adjacent experts is too large for the first k expert networks after sorting. If so, cut off the subsequent experts.

[0046] S3: The k value of hierarchical route selection after obtaining the optimal value of local route pruning under the current task and further defining it through a multi-objective optimization algorithm;

[0047] S4: In the specific large language model task inference, apply the previously obtained local route gating truncation threshold and the level k value adopted when not truncating to complete the inference optimization.

[0048] The following provides further explanation with reference to specific embodiments and execution steps:

[0049] The basis for the relative threshold pruning mentioned in step 2 is that the distribution of routing weights obtained by each token among experts at different levels is inconsistent, and there are multiple scenarios. The weights carried by the selected top k experts may be evenly distributed, while the weight percentage of the selected lower-ranked experts may have a significant difference from the previous expert. In this embodiment, the Qwen1.5-MoE-A2.7B model is used as an example. A task from the CoLA dataset was randomly selected, and the routing weight distribution of the same token in the MoE network at layers 3 and 21 was measured. Figure 2 . Figure 2 (a) The top k experts selected have relatively even weight distributions without significant disparities, making this top-k selection method trustworthy. Figure 2 (b) Among the selected top k experts, there is a significant threshold gap. A large gap indicates that the discourse weight of subsequent experts is greatly reduced, accompanied by more uncertainty, and will also lead to more computational needs. These expert networks with large gaps should not be selected, that is, the subsequent experts should be pruned.

[0050] In this type of expert pruning method for dynamic routing, the previous pruning was based on a fixed value pruning strategy. This approach faces the problem of difficulty in determining the boundary value when performing fine-grained segmentation. If the pruning is based on the percentage difference of adjacent expert activation values, the uncertainty caused by different expert segmentation methods can be reduced.

[0051] This invention utilizes the Qwen1.5-MoE-A2.7B model to randomly select a task from the CoLA dataset. During execution, the routing weights at layer 11 of the first token in the generated portion are pruned using both a fixed descent threshold and a relative descent threshold. The pruning effect is as follows: Figure 3 As shown in the figure, the weight pruning process reveals that in fine-grained expert segmentation scenarios, the fixed-value pruning method inevitably leads to the inclusion of many low-weight experts in the later selected experts as the expert activation values ​​decrease, resulting in insufficient identification. In contrast, the weight activation method based on a percentage decrease threshold can mitigate the introduction of relatively lower-weight expert networks at the tail end, reducing computational redundancy. This method is more universally applicable for selecting low-weight experts.

[0052] Cut-off criteria It can be represented as:

[0053]

[0054] Among them This represents the current input in the nth MoE layer, where the weights of the selected top-k experts are arranged in descending order. Each weight activation value. Specific The value will vary depending on the model and the task.

[0055] The final process of step 2 is as follows Figure 4 As shown, the routing is subject to two constraints: a relative decline threshold and the top-k strategy adopted by this layer.

[0056] Based on the characteristics of routing, three models with different fine-grained segmentation and routing strategies were selected: Qwen1.5-MoE-A2.7B model, OLMoE model, and Qwen3-30B-A3B model. After being trained on four different tasks, the changes in accuracy and computational cost of the models were measured according to different thresholds. The computational cost is the proportion of expert networks participating in the computation.

[0057] To verify the broad applicability of the experiment, this invention also introduced several different types of task sets, and selected some of them as evaluation vehicles for the experiment. These can be mainly categorized as GLUE, GFD, and MMLU.

[0058] The GLUE task set includes three sub-tasks: CoLA (The Corpus of Linguistic Acceptability), a single-sentence classification task with corpora from books and journals on linguistic theory, where each sentence is labeled as a sequence of words indicating whether it is grammatically correct; SST-2 (The Stanford Sentiment Treebank), a single-sentence classification task containing sentences from movie reviews and their sentiment annotations. This task assigns the sentiment of a given sentence, categorized into two classes: positive and negative sentiment, using only sentence-level labels; and QNLI (Qusetion-answering NLI), derived from the Stanford Question Answering Dataset (SQuAD 1.0), primarily used to determine the entailment relationship between questions and answers. GFD (Gearbox Fault Diagnosis Data) is a publicly available dataset shared by PIEMR (Prestige Institute of Engineering Management and Research). It uses vibration signals from different directions to detect gear faults, testing the model's adaptability to specific training tasks. From the MMLU's various task sets, three sub-tasks were selected: the abstract algebra test, the clinical knowledge test, and the high school computer science test, to test the model's understanding of the real world and its interdisciplinary problem-solving abilities.

[0059] Because the capabilities of the models vary, this embodiment differentiates them during testing. Qwen1.5-MoE-A2.7B and OLMoE have relatively weak model capabilities, requiring fine-tuning training on specific tasks to accurately identify the task target and achieve a stable task execution state. Qwen3-30B-A3B, on the other hand, has a larger number of model parameters and is more advanced; its baseline model has shown stable results on the corresponding tasks. Therefore, based on this, the test models for the task sets are also differentiated. When evaluating the GLUE and GFD datasets on Qwen1.5-MoE-A2.7B and OLMoE, fine-tuning training is performed first, followed by the execution of the corresponding optimization code. For the latest Qwen3-30B-A3B model, the GLUE and MMLU datasets are used directly for evaluation without prior fine-tuning training. Due to the specific nature of the GFD dataset, which requires specific model training, it is not included in this model's test. It also demonstrates the applicability of the method of the present invention in both cases with and without fine-tuning training for the corresponding task.

[0060] The seven selected tasks aim to test the method of this invention in different application environments, measuring the changes in the model's computational speed and capabilities from four aspects: grammatical, sentiment, semantic understanding, and target detection. This embodiment conducted relevant relative threshold pruning experiments for these seven tasks and three models.

[0061] like Figure 5 As shown, it is evident that the model's adaptability to the threshold varies across different tasks, and the level of model granularity also affects the proportion of expert redundancy. However, the pruning threshold is consistently around 0.6. Overall, an appropriate pruning threshold can not only reduce the model's computational burden but may even improve accuracy. This indicates that redundant non-experts do indeed have a negative impact on model inference, meaning that concentrated inference by expert members can benefit the model's accuracy.

[0062] For the global multi-objective optimization problem in step 3, based on this global optimization choice problem, this invention introduces the Fast Non-Dominated Sorting Genetic Algorithm II (NSGA-II), which searches for the k-value of each layer of experts suitable for the current task in a fixed test task. Compared with traditional static or gradient-driven optimization methods, the heuristic algorithm explores the parameter space through flexible rules, achieving fast response with low computational overhead. Moreover, its non-gradient characteristic can avoid the aggregation parameters from getting trapped in local optima. Especially under the conditions of weak interpretability and emergence capability limitations in LLM structures, it can quantify the benefits brought by the selection of levels through adaptive k-values.

[0063] This invention selects NSGA-II as the solution algorithm for MoE architecture optimization, primarily based on its widespread application and mature performance in the field of multi-objective optimization. As a classic multi-objective evolutionary algorithm, NSGA-II, through non-dominated sorting and crowding distance mechanisms, can maintain the diversity of the solution set while ensuring population convergence, thus effectively approximating the Pareto optimal front. Existing research shows that NSGA-II has the characteristics of simple algorithm structure, fewer parameter settings, a sound elite retention mechanism, and good solution stability, and is therefore widely used in engineering optimization, resource allocation, and machine learning-related multi-objective decision problems. At the same time, the field of multi-objective optimization has also developed a number of representative algorithms, including SPEA2 and MOEA / D, which improve solution performance from different perspectives such as Pareto dominance and decomposition strategies. However, a large number of studies have pointed out that the performance of multi-objective evolutionary algorithms usually depends on the specific problem characteristics, the properties of the objective function, and the decision space structure. No single algorithm can consistently outperform other methods in all scenarios. Therefore, this invention selects NSGA-II not based on its absolute advantage over other heuristic algorithms, but rather on its mature theoretical foundation, sufficient application verification, and good adaptability to multi-objective optimization tasks.

[0064] The population iterated by the NSGA-II algorithm represents the populations at each level of the MoE architecture in the current task. The value can be represented as:

[0065]

[0066] For this type of multi-objective optimization problem, the solution Dominant Solution If and only if:

[0067]

[0068] Where M is the number of objective functions. and These represent different objective functions.

[0069] In NSGA-II, a fast non-dominated sort is first performed on the merged population: for each individual, the number of individuals dominating it is counted and the solution set it dominates is recorded. Then, individuals with zero dominant individuals are used as the first frontier, and the dominated count is recursively reduced layer by layer to divide the subsequent frontiers. This process completes the entire rank division in linear time complexity, ensuring that the population can be efficiently divided into multiple Pareto ranks. When selecting individuals to enter the next batch of optimized populations, most of the time all individuals in the first n ranks are included, and only a portion of the (n+1)th rank is included. The selection of which individuals in the (n+1)th rank are included in the offspring is determined by a crowding-based sorting method.

[0070] Crowding is used to measure how dense each individual is within its non-dominated layer. An individual with higher crowding indicates a sparser population of surrounding individuals and a greater likelihood of being selected for the next generation. The crowding of each individual is initialized. Crowding distance is used to measure the degree of crowding in the solution, and the individuals in this layer are ranked according to the objective function value. Sort the solutions to determine which one should win in the binary tournament.

[0071]

[0072] in, and These are the maximum and minimum values ​​of the t-th objective function in this layer, respectively.

[0073] Specifically, the NSGA-II algorithm's population propagation and updates rely on a combination of an elite preservation strategy and genetic operators. In the t-th generation of evolution, the parent population, each with a size of N, is first... With offspring population The populations were merged to form a mixed population of size 2N. :

[0074]

[0075] This elite preservation mechanism ensures that the historical optimal solution is not lost in the evolutionary process. Environmental selection then follows. Performing the aforementioned fast non-dominated sorting yields multiple Pareto fronts. The algorithm sequentially... Introducing a new generation of parental populations until the critical layer The addition of [a certain element] will cause the population size to exceed N. At this point, [the population size will be larger than N]. Within a layer, selection is based on distance according to congestion level; this process is performed by a congestion level comparison operator. Strict definition. For any two individuals i and j, individual i is superior to individual j if and only if the following condition is satisfied:

[0076]

[0077] When individuals are at different non-dominated layers, priority is given to those with higher frontier ranks and lower rank values. If two individuals are at the same frontier (i.e., with the same rank value), priority is given to those with sparser surrounding environments and greater crowding distances, thus selecting the best individuals to fill the gaps. N positions.

[0078] In generating a new generation of offspring populations At that time, the algorithm passed the binary tournament from Select a superior parent and perform crossover and mutation. For numerical problems involving MoE architecture optimization, simulated binary crossover (SBX) and polynomial mutation (PM) are typically used. Let the two selected parent individuals be... and Offspring individuals generated by SBX crossover , It can be calculated using the following formula:

[0079]

[0080] in The expansion factor, determined by the distribution index, is used to simulate the probability distribution of a single-point crossover. Next, multinomial mutations are performed on the crossover individuals y with a certain probability to fine-tune the gene loci, generating the final mutated individuals. :

[0081]

[0082] in Let be the variation perturbation factor of the multinomial distribution. This dimension represents the span of the value space for the MoE level parameters. In this way, the algorithm not only inherits the superior characteristics of its parent generation but also ensures the diversity of the population in the search space.

[0083] In this way, the algorithm not only inherits the superior characteristics of its parents but also ensures the diversity of the population in the search space. After classic genetic algorithm processes such as crossover, mutation, merging, and environmental selection, the non-dominated solution set of the final generation, i.e., the approximate Pareto front solution set, is generated. This solution set is a one-dimensional tensor representing the k-values ​​at each level, balancing the model's task accuracy and computational cost gains, allowing for an intuitive analysis of the optimization algorithm's gains relative to the original experiment at both levels.

[0084] The objective function's value originates from the model's accurate derivation within a specific task domain and the computational gains during the process. This dictates that during algorithm implementation, the model needs to continuously receive iterative population values ​​from the algorithm, evaluate them within the task, and use these values ​​as feedback to the objective function for algorithm optimization. The implementation process of the above steps is as follows: Figure 6 As shown.

[0085] To measure the algorithm's effectiveness, two objective functions are introduced as directions for model optimization. The first is an accuracy function for judging the model's capabilities, which uses the percentage of correct answers performed by the large model in relevant inference tasks as the optimization objective. This is represented by the function F, which describes the inference process of the large model. It can be expressed as:

[0086]

[0087] in This represents the solution in the currently selected population, while That is, the number of task sets.

[0088] After iterations of NSGA-II, a Pareto front solution set based on two objective functions is obtained. This is used to explore the impact of the model on the k-bound of each MoE layer in different tasks, and to derive task-specific k-values. The aim is to reduce the computational load of the model while maintaining its accuracy. It is observed whether the professionalism of the model inference process will improve or the model's ability will decrease due to the reduction in the number of expert networks.

[0089] In the experiment, a one-dimensional tensor, whose length is equal to the number of network layers in the model, was constructed using the k-value of each layer. The k-values ​​were fixed between 2 and 4, and between 2 and 7, respectively, to avoid extreme cases of maximum and minimum values, adapting to the top-4 routing of Qwen1.5-MoE-A2.7B and the top-8 routing of OLMoE and Qwen3-30B-A3B models. The accuracy evaluation of the large model on the corresponding task and the reduction of computational cost were used as objective functions to iteratively optimize the population, aiming to find the Pareto front of the population, i.e., the pattern of k-values ​​at each layer in the current task domain.

[0090] Through iterative optimization of the population in the NSGA-II algorithm, the final Pareto front was obtained for the same task. The optimal k values ​​at multiple levels showed general similarity, as illustrated in the specific distribution heatmap. Figure 7 As shown, only in a few levels will different Pareto fronts have prominent values, which leads to differences in accuracy and computational cost. In different task domains, there are indeed patterns in the k value for each level, with a set of relatively optimal k values ​​that can bring benefits to both objective functions. The k value requirements for different levels will also be different for each task, but observations show that some levels place more emphasis on a k value of 3, such as levels 4, 6, 7, 8, and 21 in the CoLA task, where all Pareto solution sets are consistent, while a k value of 4 is considered a waste of resources in most cases.

[0091] Step 4 involves applying the optimized model based on the Pareto front of the optimal cutoff threshold and level k value obtained for different models in different tasks, rather than seeking the same values ​​for all models and all tasks.

[0092] Based on the results obtained in steps 2 and 3, end-to-end measurements were performed in this embodiment, such as... Figure 8 As shown, the average processing time per token for the two MoE network models is recorded across four datasets. Figure 8 As shown, the average processing time per token for most networks in the model is reduced by approximately 30% to 60%. Since the introduced fast non-dominated sorting genetic algorithm also introduces some additional data processing, the computation time does not strictly correspond to the reduction in computational cost, but the trend and magnitude of the reduction are roughly the same as the pruning results presented in the previous heatmap.

[0093] In summary, this invention can effectively identify expert networks with redundant computations in different task applications of different models, achieving significant benefits in terms of saving computational costs and improving processing speed.

[0094] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A hybrid routing optimization method based on a large MoE architecture model, characterized in that, Includes the following steps: Step 1: Extract the expert weights after passing through the gating matrix in the routing section of the MoE network; Step 2: Determine whether there is a situation where the relative difference between the weights of adjacent experts is too large after sorting the first k expert networks. If so, cut off the subsequent experts. Step 3: After obtaining the optimal value of local route pruning under the current task, the k value of hierarchical route selection is further defined by a multi-objective optimization algorithm; Step 4: Apply the previously obtained local routing gating truncation threshold and the level k value used when not truncating in the specific large language model task inference to complete the inference optimization.

2. The hybrid routing optimization method based on the MoE architecture large model as described in claim 1, characterized in that, The expert weights extracted in step 1 are vectors representing the weights of all experts in the large model at the current token batch in the decoder stage, obtained by passing the intermediate vector through the gate matrix of the MoE network. The length of the vector is related to the batch size of the tokens and the preset number of experts in the selected model.

3. The hybrid routing optimization method based on the MoE architecture large model as described in claim 1, characterized in that, Step 2, in determining cases where the relative difference in weights between adjacent experts is too large, specifically involves calculating the relative proportion of the weight decrease from one expert to the next in the descendingly sorted experts. The truncation criterion is expressed as follows: in, This represents the current input in the nth MoE layer, where the weights of the selected top-k experts are arranged in descending order. Individual weighted activation values; specific thresholds The value will vary depending on the model and the task.

4. The hybrid routing optimization method based on the MoE architecture large model as described in claim 1, characterized in that, The multi-objective optimization algorithm used in step 3 is the NSGA-II algorithm. The objective function is to maximize the task accuracy while minimizing the number of experts used by the model. The input is a one-dimensional vector whose length corresponds to the number of layers in the model. Each element represents the top-k value adopted by that layer.

5. The hybrid routing optimization method based on the MoE architecture large model as described in claim 1, characterized in that, The multi-objective optimization algorithm is based on step 2 and is responsible for determining the optimal value of the route truncation threshold at the local level, and defining the global level k value in subsequent algorithm optimization.

6. The hybrid routing optimization method based on the MoE architecture large model as described in claim 4, characterized in that, The multi-objective optimization process includes the following steps: Step 3.1: First, the algorithm generates an initial population, inputs the round value into the evaluation model, and allows the model to implement the corresponding k value at each layer in the specific task, thereby obtaining the task accuracy and expert usage rate of the evaluation. Step 3.2: Perform non-dominated sorting and crowding calculation on the objective function results of the initial population to generate the first generation parent population; Step 3.3: From the current parent population, generate a child population of the same size by calculating the sorting level and crowding degree, crossover, and mutation genetic operators; Step 3.4: Merge the parent and offspring populations to form a temporary population of size 2N. Perform fast non-dominated sorting, and then select N individuals based on sorting hierarchy and crowding to form the new generation of parent population. Step 3.5: Use the newly generated parent population as the parent population for the next iteration, return to step 3.3, and continue until the termination condition is met, i.e., the maximum number of iterations is reached, or the solution set no longer improves significantly.

7. The hybrid routing optimization method based on the MoE architecture large model as described in claim 1, characterized in that, Step 4 involves applying the optimized model based on the Pareto front of the optimal cutoff threshold and level k value obtained for different models in different tasks, rather than seeking the same values ​​for all models and all tasks.

Images