Hybrid expert model training optimization method based on matrix routing and Token allocation

Abstract

The invention discloses a mixed expert model training optimization method based on matrix routing and Token allocation, and the method comprises the steps: firstly, in a pre-training process, an MoE model processes different input data through a plurality of expert sub-models, generates an expert index through a gating network, and generates a scheduling index and a combined index based on the expert index; secondly, based on the generation of the scheduling index and the combined index, entering a token scheduling and combined token stage, scheduling the token to each expert for expert processing, and combining the tokens processed by the experts by the combined token stage to generate a combined token; and finally, after pre-training of the MoE model is completed, fine tuning is carried out, and training optimization is completed. According to the method, on the premise that the fine adjustment precision of the model is kept unchanged, expert computing resources are fully utilized, and resource idling is effectively avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network, and in particular relates to a hybrid expert model training optimization method based on matrix routing and Token allocation. Background Art

[0002] In recent years, large models have shown excellent performance in tasks such as natural language processing (NLP), computer vision (CV), and recommendation systems. Expanding the model size has been seen as a key technical means to further improve model performance. The scale of model parameters has also rapidly increased from hundreds of billions to trillions, such as Llama 2, GPT4, etc. However, the training and reasoning of these ultra-large-scale models require extremely high computing and memory resources. For example, GPT-3 has 175 billion parameters and costs more than $1.2 million to train on 1,000 A100 GPUs for 22 days. Therefore, in order to reduce the training computational cost of the model while expanding the model size, the mixed expert model (MoE) technology has received widespread attention from AI practitioners. MoE significantly reduces the model's computational requirements by activating only some of the model parameters, and has been successfully applied to models such as Mixtral and Grok-1.

[0003] The MoE model is a complex neural network structure that uses multiple specialized sub-models (experts) to process different input data. During the pre-training process of the MoE model, the input token is selected by the gating network to activate the indexes of the first k experts and realize the scheduling of tokens to experts. The scheduling stage generates a scheduling index, which is used to schedule the token to the corresponding expert, and each expert calculates it separately. After that, a combination index is generated in the combination stage, and the output of the experts is aggregated based on these combination indexes to obtain the final result. However, when analyzing the pre-training and fine-tuning stages of MoE, it was found that there are two major challenges in terms of computational overhead and accuracy. First, in the pre-training stage, the generation of scheduling index and combination index leads to a lot of computational overhead. The expert index (DP, N) needs to obtain the scheduling index and combination index through the addition, subtraction, multiplication and division operations of multiple high-dimensional matrices. The matrix dimension is (DP, N, E*C), and the time complexity reaches O(N^3). Secondly, during the scheduling and combination operations, the scheduling and combination of tokens requires complex matrix calculations. For example, in GShard, Switch Transformers, and Mixtral, the scheduling index needs to be multiplied by a high-dimensional matrix: [DP, H, N] * [DP, N, E * C] = [DP, H, E * C], which greatly increases the computational cost and the time complexity reaches O (N^3). Finally, in the fine-tuning stage, token dropping is often used to alleviate the overfitting problem of MoE, but small sample data sets contain a large number of zero vectors. The token allocation methods such as GShard, Switch Transformers, Tutel, and Mixtral are not appropriate in the fine-tuning stage, resulting in a large amount of computing and memory resources being wasted, resulting in a contradiction between accuracy and hardware resource requirements, especially under the top-k (k ≥ 2) selection strategy. Therefore, in view of the computational efficiency and accuracy challenges of the MoE model in the pre-training and fine-tuning stages, it is particularly important to optimize the model's computational methods and resource allocation. Summary of the invention

[0004] The present invention mainly addresses two key problems of the MoE model in the pre-training and fine-tuning stages: high computational overhead and precision loss, and proposes a hybrid expert model training optimization method based on matrix routing and Token allocation to solve the above problems.

[0005] The present invention provides a hybrid expert model training optimization method based on matrix routing and Token allocation, and the analysis method is as follows:

[0006] Step 1: During the pre-training process, the MoE model uses multiple expert sub-models to process different input data. In this process, the MoE model first needs to generate expert indexes through a gating network (mainly composed of softmax functions), and then generate scheduling indexes and combination indexes based on the expert indexes. During the scheduling index generation process, each token selects the corresponding expert based on the gating network, and summarizes the expert selection results of each token into a two-dimensional matrix M. This matrix M is implemented by the following formula:

[0007]

[0008] Among them, M ij is the element in the i-th row and j-th column of the matrix M, token i For the i-th token, expert j In order to reduce the impact of unselected experts on the priority in subsequent processing, a matrix X of the same size as matrix M is constructed, and the matrix M and matrix X (the elements in the matrix are much larger than each element in M) are calculated as follows to make the non-selected positions have a lower priority:

[0009] M′=range(N)*M+(X+range(N))*(1-M)

[0010] Where N represents the length of X, and range represents a loop function, which is generated from 0 to N-1. In the generated matrix M′, non-zero vector values ​​are extracted through quick sorting and slicing, and then all indexes greater than or equal to X are reset to -1, indicating that the token is not processed by the expert, thereby completing the generation of the scheduling index, and the time complexity is reduced to O(Nlog(N)). The scheduling index is a three-dimensional matrix with dimensions (DP, E, C), DP is the dimension of data parallelism, E is the number of experts, and C is the size of the expert capacity.

[0011] For combined index generation, the algorithm calculates the cumulative sum Cumsum along the row dimension based on the matrix M through the parallel matrix processing characteristics of the GPU, keeping the position of the selected experts unchanged:

[0012] H=Cumsum(M,axis=1)×M

[0013] Add the sequence vector V of the corresponding expert to each column of the matrix H. The vector V is generated by the following formula:

[0014] V=Q×C

[0015] Q is a one-dimensional column vector, each element is the number of rows minus 1, C is the expert capacity of MoE, indicating the number of tokens processed by each expert; after obtaining H, sum each column and subtract 1 to get the combined index. This process reduces the time complexity of generating the combined index from O(N^3) to O(1), greatly improving the computing efficiency. The scheduling index is a two-dimensional matrix with dimensions (DP, N).

[0016] Step 2: After generating the scheduling index and the combination index, it is necessary to enter the scheduling token and combination token stage. The former will schedule the token to each expert for expert processing. After the expert processing, the token will be combined to generate a combination token. In the scheduling process and the combination process, it is necessary to search for the physical address of the token based on the scheduling index and the combination index generated in step 1, so as to realize the scheduling and combination operations. This process usually requires the use of a high-dimensional matrix multiplication, and the time complexity reaches O(N^3). To this end, the processing of logical indexes is optimized by using the parallel computing capability of the GPU, which can significantly reduce the complexity. Specifically, the logical index includes two types: scheduling index and combination index. Based on the above information, each thread on the GPU runs independently, directly calculates the corresponding physical address from the logical index, and completes data migration and writing. Since there are two forms of logical indexes, each thread needs to independently calculate a unified target index based on the logical index. The present invention will first use the unified computing device architecture CUDA thread structure to obtain the token information that each thread block needs to process. First, an index write_index will be obtained, and its calculation formula is:

[0017] write_index=blockIdx.x*blockDim.x+threadIdx.x

[0018] +w×gridDim.x×blockDim.x

[0019] Among them: blockIdx.x, blockDim.x, threadIdx.x and gridDim.x are the basic parameters of the CUDA thread structure; w is a loop variable, which represents the Token block processed by the current thread. Based on the index write_index, the multi-dimensional space coordinates of the data can be parsed.

[0020]

[0021] k=writr_index mod output_dim2

[0022] Among them, output_dim0, output_dim1, and output_dim2 represent the logical index information of the input. For example, for the scheduling index (DP, E, C), each dimension represents an input. When the dimension does not exist, it will be set to the default value of 1. In addition, i corresponds to the expert dimension, j corresponds to the token dimension, and k represents the specific feature dimension. The calculation of the physical address will be based on the three dimensions i, j, and k and the position of this token in the expert index (DP, N) (referred to as indices). The formula is as follows:

[0023] read_index=i*ouput_dim1*output_dim2+indices*output_dim2+k Through this formula, each thread independently completes the mapping from logical index to physical address, ensuring that data can be efficiently migrated and obtaining a unified target index read_index. After each thread calculates the physical address based on the target index, it will be used to write data at the target location:

[0024] output[write_index]=input[read_index]

[0025] Since each thread completes an index calculation and data migration independently, the time complexity of the operation is reduced from O(N^3) to O(1), greatly reducing the time spent on physical address lookup.

[0026] Step 3: After completing the pre-training of the above-mentioned MoE (mixed expert) model, the model enters the fine-tuning stage to complete the training optimization. At this time, it usually faces two important challenges: the existence of zero vectors and the reasonable allocation of tokens. In response to these problems, the present invention proposes an optimized token partitioning and allocation method to improve resource utilization efficiency and ensure the fine-tuning accuracy of the model. In the fine-tuning stage, the proportion of zero vectors in small sample data may be high. Zero vectors are usually some inputs that lack valid information, and excessive processing of these zero vectors will lead to a waste of computing resources. In order to avoid this situation, it is necessary to analyze and divide the input tokens, and this part of the data is mainly divided into two parts: a valid queue and an invalid queue. The valid queue contains all real tokens with non-zero importance values. The invalid queue contains zero vectors, which are used as fill-in backup first, but will not directly participate in expert processing.

[0027] After completing the division of valid and invalid queues, tokens need to be efficiently allocated to MoE experts. The goal here is to prioritize valid tokens while maximizing resource utilization to avoid idle computing resources. Using the Top-k strategy (each token is assigned to a maximum of k experts), sort the tokens in the valid queue by importance and assign them to experts in order of importance. After valid tokens are allocated, some experts may still have idle computing positions. In order to keep all experts processing the same number of tokens, tokens in the invalid queue need to be used to fill these vacancies. This method ensures that real valid tokens are processed first and ensures the accuracy of the model.

[0028] The beneficial effects of the present invention are:

[0029] The present invention proposes several optimization methods to balance the computational efficiency and accuracy of the sparse expert model (MoE) during pre-training and fine-tuning, which brings significant technical advantages. First, by scheduling the generation algorithm of indexes and combined indexes, the time complexity of the original three-dimensional matrix multiplication is reduced from O(n^3) to O(Nlog(N)), thereby achieving efficient index generation. Secondly, the present invention utilizes a novel addressing method to further optimize the search complexity of the token physical address from O(N^3) to O(1), making full use of the parallel processing capability of the GPU and greatly improving the data access speed. In addition, in the token allocation of the top-k strategy, the present invention introduces a new token allocation method, which changes the dynamic allocation method of tokens. While ensuring that all valid tokens are processed, invalid tokens are discarded to ensure that expert computing resources are fully utilized while the model fine-tuning accuracy remains unchanged, effectively avoiding resource idleness. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is the scheduling index generation algorithm graph;

[0031] Figure 2 It is a graph of the combined index generation algorithm;

[0032] Figure 3 It is an optimized design diagram for token distribution. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation steps. A hybrid expert model training optimization method based on matrix routing and Token allocation includes the following process:

[0034] Step 1: First, during the pre-training process, the token will be distributed to which expert in the MoE model through the gating network. After the expert processes the token, it will be recombined to generate the corresponding output content. This process first generates a scheduling index and a combination index to implement the token scheduling and combination process. Figure 1 In the process of generating the scheduling index, the present invention completes the generation of the scheduling index through the following four steps.

[0035] Step 1.1: Each token gets the expert selection of each token based on the gated network, that is Figure 1 As shown in step (a) of , after obtaining the corresponding expert index, a one-dimensional vector is formed, and these vectors are aggregated into a two-dimensional matrix M recorded as (DP, N), as shown in step (b). As shown in the following formula:

[0036]

[0037] Step 1.2: After obtaining the matrix M, modify each non-zero element to obtain the number of valid elements before the column where the element is located, and assign the value to the element, as shown in step (c). Then, the present invention combines the matrix M with the range matrix and adds a maximum value X(X) to the unselected expert position.

[0038] N) to ensure that they have lower priority in subsequent steps, as shown in step (d), with the following formula:

[0039] M′=range(N)*M+(X+range(N))*(1-M)

[0040] Where range(N) generates a sequential matrix, N represents the number of columns of M, and X is used to reduce the priority of non-selected positions. After generating the matrix M′, quick sort and slicing operations are used. The slicing of the slicing operation is mainly determined by the expert capacity. As shown in steps (e) and (f), the non-zero vector values ​​are extracted.

[0041] Step 1.3: Reset all indexes greater than or equal to X to -1, indicating that the token will not be processed by the expert, thereby completing the generation of the scheduling index, where the matrix dimension of the scheduling index is (DP, E, C), DP is the dimension of data parallelism, E is the number of experts, and C is the size of the expert capacity.

[0042] The overall implementation method is shown in Algorithm 1. The whole process uses the quick sort algorithm to reduce the time complexity of generating the scheduling index from O(N^3) to O(NlogN).

[0043]

[0044] Step 2: For composite index generation, the process is as follows Figure 2 shown.

[0045] Step 2.1: This step reuses the matrix M used in the process of generating the scheduling index, and the matrix dimension is also (DP, N). Secondly, the invention replaces the target element based on the row value. The replacement rule is to calculate the cumulative sum of the matrix M based on the row dimension, and multiply it with the original matrix to ensure that the position of the expert remains unchanged, as shown in step (c). The formula is as follows:

[0046] H=Cumsum(M,axis=1)×M

[0047] After replacing the value of each element, it is necessary to add the sequence vector V of the corresponding expert to each column of the matrix H. The vector V is generated by the following formula:

[0048] V=Q×C

[0049] Q is a one-dimensional column vector, each element of which is the number of rows minus 1, and C is the expert capacity of MoE, which indicates the number of tokens that each expert can process.

[0050] Step 2.2: After obtaining H, the present invention will sum each column and subtract 1 to obtain a combined index, as shown in the following formula:

[0051] CombineIndex=ReduceSum(C,axis=0)-1

[0052] Algorithm 2 describes the process in detail. By taking advantage of the parallel matrix processing characteristics of the GPU accelerator, the time complexity is reduced from O(N^3) to O(1), effectively improving the computational efficiency.

[0053]

[0054] Step 3: After completing the generation of the scheduling index and the combination index, it is necessary to enter the scheduling token and combination token stage. The scheduling token stage is responsible for assigning each token to the corresponding expert for processing, while the combination token stage integrates the output results of each expert. In these two stages, the core task is to efficiently find the physical address of the token based on the logical index (scheduling index and combination index) generated by steps 1 and 2 to complete the scheduling and combination of data. However, the traditional method implements physical address search through high-dimensional matrix multiplication, with a time complexity of up to O(N^3), which is inefficient in large-scale tasks.

[0055] To this end, the present invention utilizes the parallel computing capability of the GPU and directly processes the logical index to significantly optimize the address calculation process, reducing the complexity from O(N^3) to O(1). Figure 3 As shown, the specific method is as follows:

[0056] Step 3.1: In the CUDA architecture, each thread processes a Token block independently. Based on the GPU thread structure, the global index write_index of each thread can be calculated to determine the Token that the thread needs to process. The formula is:

[0057] write_index=blockIdx.x*blockDim.x+threadIdx.x

[0058] +w×gridDim.x×blockDim.x

[0059] Among them: blockIdx.x, blockDim.x, threadIdx.x, gridDim.x are the basic parameters of the CUDA thread structure; w is the loop variable, which represents the Token block processed by the current thread. Through this formula, each thread clearly defines its own task scope in the global calculation.

[0060] Step 3.2: Based on write_index, the thread further parses the multidimensional space coordinates of the Token in the logical index. Assuming that the dimensions of the logical index are output_dim0, output_dim1, and output_dim2, the parsing formula is as follows:

[0061]

[0062] k=write_index mod output_dim2

[0063] Where: i represents the expert dimension; j represents the token dimension; k represents the feature dimension. For the scheduling index (DP, E, C) and the combination index (DP, N), when some dimensions are missing, the default value can be set to 1.

[0064] Step 3.3: Through the logical coordinates i, j, k and the position indices of the token in the expert index, the physical address is calculated as follows:

[0065] read_index=i*output_dim1*output_dim2+indices*output_dim2+k, where output_dim1 and output_dim2 represent the dimension size of the logical index.

[0066] Step 3.4: After each thread completes the calculation of the physical address, it directly extracts the content from the input data and writes it to the target address:

[0067] output[write_index]=input[read_index]

[0068] This method reduces the time complexity of physical address lookup and data migration from O(N^3) to O(1) by performing independent calculations on threads, significantly improving the efficiency of the scheduling and combination stages, and providing a key optimization means for the application of the MoE model in large-scale computing scenarios.

[0069] Step 4: After completing the pre-training of the MoE (mixture of experts) model, a major challenge will be faced when entering the fine-tuning stage: the existence of zero vectors leads to unreasonable token allocation. To this end, the present invention proposes an optimized token allocation method based on the division of valid queues and invalid queues, aiming to maximize resource utilization efficiency and ensure the accuracy of model fine-tuning.

[0070] Step 4.1: The data in the fine-tuning stage often contains a large number of zero vectors (invalid inputs), which lack effective information. If they are directly involved in the calculation, it will not only waste computing resources but also affect the performance of the model.

[0071] Therefore, the input data needs to be divided through the following steps:

[0072] Importance detection: The i-th input token is represented by T i , calculate its importance value I(T i ). The importance value can be calculated by the following formula:

[0073] I(T i )=|T i |2

[0074] Where |T i |2 represents the L2 norm of the token. When I(T i )=0, indicating that T i is the zero vector.

[0075] The queue is divided into valid queue and invalid queue. The valid queue represents all I(T i )>0 (i.e., tokens with non-zero importance values). The invalid queue represents all I(T i )=0. Through the above method, it is ensured that all real and valid tokens are identified and divided into the valid queue, while invalid tokens (zero vector) enter the spare invalid queue.

[0076] Step 4.2: After completing the division of valid queues and invalid queues, tokens need to be assigned to experts in the MoE model for processing. Based on the Top-k strategy (each token is assigned to at most k experts), valid tokens are assigned first. Assume that the tokens in the valid queue are {T1, T2, …, T m}, there are m valid tokens in total. i ) for {T1,T2,…,T m} to sort in descending order:

[0077] I(T1)≥I(T2)≥…≥I(T m )

[0078] After sorting is completed, for each T i , and are distributed to k experts with idle computing resources in turn to ensure that these valid tokens can be processed first.

[0079] Step 4.3: After all real tokens are allocated, the vacant positions in each expert are filled with zero vectors. This filling of zero vectors ensures that the number of tokens processed by all experts remains consistent, so as to achieve efficient GPU parallel effect. According to the allocation of tokens, the present invention adjusts the load of tokens according to the capabilities of each expert, so that each expert gets the content that needs to be processed. This systematic zero vector allocation ensures a balance between computational efficiency and model accuracy, and ensures the optimal load of each expert through balanced allocation.

[0080] The experimental results are shown in Table 1 and Table 2. Compared with Switch Transformers, this embodiment verifies the performance of the invention on various billion-level, tens of billions-level, and hundreds of billions-level models and three datasets: Wikitext, C4, and Lambada. It is concluded that the overall performance of the present invention is better than that of Switch Transformers, and when applied to Mixtral and GPT-MoE models, the present invention can achieve acceleration ratios of 1.39 times and 1.20 times, respectively. In terms of scalability, the present invention verifies scalability through different model parameters. Compared with Switch Transformers, as the model parameter size and the number of GPU accelerators increase, the present invention maintains stable acceleration performance. This shows that the present invention is more conducive to application to larger models.

[0081] Table 1: Speedup effect of Mixtral model in different scenarios

[0082]

[0083] Table 2: Speedup effect of GPT-MoE model in different scenarios

[0084]

[0085]

Claims

1. A hybrid expert model training optimization method based on matrix routing and token allocation, characterized in that: The steps include: Step 1: During the pre-training process, the MoE model uses multiple expert sub-models to process different input data. The MoE model generates an expert index through a gating network, and generates a scheduling index and a combination index based on the expert index. Step 2: Based on the generated scheduling index and combination index, enter the scheduling token and combination token stage, schedule the token to each expert for expert processing, and combine the tokens after expert processing to generate a combination token; Step 3: After completing the pre-training of the above MoE model, fine-tune it to complete the training optimization.

2. The hybrid expert model training optimization method based on matrix routing and Token allocation according to claim 1 is characterized in that: The generation process of the scheduling index is as follows: each token selects the corresponding expert based on the gating network, and the expert selection results of each token are summarized into a two-dimensional matrix M. The matrix M is implemented by the following formula: Among them, M ij is the element in the i-th row and j-th column of the matrix M, token i For the i-th token, expert j Denotes the jth expert; Construct a matrix X of the same size as matrix M. The elements in matrix X are much larger than each element in M. Calculate matrix M and matrix X as follows: M′=range(N)*M+(X+range(N))*(1-M) Where N represents the length of X, range represents a loop function, generating from 0 to N-1; in the generated matrix M′, non-zero vector values ​​are extracted through quick sorting and slicing, and then all indexes greater than or equal to X are reset to -1, indicating that the token is not processed by the expert, and the generation of the scheduling index is completed, and the time complexity is reduced to O(Nlog(N)); the scheduling index is a three-dimensional matrix with dimensions (DP, E, C), DP is the dimension of data parallelism, E is the number of experts, and C is the size of the expert capacity.

3. The hybrid expert model training optimization method based on matrix routing and Token allocation according to claim 2 is characterized in that: The process of generating the combined index is as follows: based on the matrix M, the cumulative sum Cumsum is calculated along the row dimension through the parallel matrix processing characteristics of the GPU, and the positions of the selected experts are kept unchanged: H=Cumsum(M,axis=1)×M Add the sequence vector V of the corresponding expert to each column of the matrix H. The vector V is generated by the following formula: V=Q×C Q is a one-dimensional column vector, each element is the row number - 1; After obtaining the matrix H, sum each column and subtract 1 to get the combined index; this process reduces the time complexity of generating the combined index from O(N^3) to O(1), where the scheduling index is a two-dimensional matrix with dimension (DP, N).

4. The hybrid expert model training optimization method based on matrix routing and Token allocation according to claim 3 is characterized in that: The step 2 also includes: in the process of scheduling tokens and combining tokens, the physical address of the token is searched based on the scheduling index and the combination index generated in step 1. This process optimizes the processing of logical indexes by using the parallel computing capability of the GPU; the logical index includes two types: scheduling index and combination index. Each thread on the GPU runs independently, and the corresponding physical address is directly calculated from the logical index, and data migration and writing are completed; there are two forms of logical indexes, and each thread independently calculates a unified target index based on the logical index.

5. The hybrid expert model training optimization method based on matrix routing and Token allocation according to claim 4 is characterized in that: Each thread independently calculates a unified target index based on the logical index. The specific process is as follows: Using the CUDA thread structure to obtain the token information processed by each thread block, first obtain an index write_index, which is calculated as follows: write_index=blockIdx.x*blockDim.x+threadIdx.x+w×gridDim.x×blockDim.x Among them: blockIdx.x, blockDim.x, threadIdx.x and gridDim.x are the basic parameters of the CUDA thread structure; w is the loop variable, which represents the Token block processed by the current thread; based on the index write_index, the multidimensional space coordinates of the data are parsed: k=write_indexmodoutput_dim2 Among them, output_dim0, output_dim1, and output_dim2 represent the logical index information of the input; when the dimension does not exist, it is set to the default value 1; in addition, i corresponds to the expert dimension, j corresponds to the token dimension, and k represents the specific feature dimension; the calculation of the physical address will be based on the three dimensions i, j, k and the position indices of this token in the expert index (DP, N), and the formula is as follows: read_index=i*ouput_dim1*output_dim2+indices*output_dim2+k Through this formula, each thread independently completes the mapping from logical index to physical address and obtains a unified target index read_index. After each thread calculates the physical address based on the target index, it will be used to write data at the target location: output[write_index]=input[read_index] Each thread independently completes an index calculation and data migration, reducing the time complexity of the operation from O(N^3) to O(1).

6. The hybrid expert model training optimization method based on matrix routing and Token allocation according to claim 5 is characterized in that: The specific implementation process of the fine-tuning is as follows: Analyze and divide the input tokens. This part of data is mainly divided into two parts: valid queue and invalid queue. The valid queue contains all real tokens with non-zero importance values; the invalid queue contains zero vectors, which are used as fillers and are not directly involved in expert processing; After completing the division of valid queues and invalid queues, tokens are allocated to MoE experts. Using the Top-k strategy, each token is allocated to at most k experts. The tokens in the valid queue are sorted by importance and allocated to experts in order of importance. After the valid tokens are allocated, if there are experts with idle computing positions, the vacancies will be filled with tokens in the invalid queue.

7. The hybrid expert model training optimization method based on matrix routing and Token allocation according to claim 6 is characterized in that: The division of the valid queue and the invalid queue is specifically as follows: The i-th input token is denoted as T i , calculate its importance value I(T i ): I(T i )=|T i |2 Where |T i |2 represents the L2 norm of the token; when I(T i )=0, indicating that T i is the zero vector; The effective queue represents all I(T i )>0 token; Invalid queue means all I(T i )=0.

Images