MoE large model reasoning optimization method and device based on expert module dynamic scheduling

Abstract

The invention discloses a MoE large model reasoning optimization method and device based on expert module dynamic scheduling, and relates to the technical field of large model reasoning optimization. The method comprises the steps of performing pre-reasoning through a MoE large model according to a dialogue text data set to obtain a first evaluation score set; performing value optimization on the GPU candidate expert quantity value set through an expert cache hit rate calculation method to obtain a GPU candidate expert optimized value set; cPU expert module dynamic scheduling is carried out based on the relation curve of the expert score ascending value-the transmission expert number, and a CPU expert scheduling index list is obtained; gPU expert module dynamic supplementation is carried out according to the first evaluation score set and the CPU expert scheduling index list, a GPU expert supplementation index list is obtained, re-reasoning is carried out through a MoE large model, and a reasoning result text data set is obtained. The invention relates to a large model reasoning optimization method for balancing model reasoning speed and output result quality of a MoE large model.

Description

Technical Field

[0001] This invention relates to the field of large model inference optimization technology, and in particular to a method and apparatus for MoE large model inference optimization based on dynamic scheduling of expert modules. Background Technology

[0002] With the continuous development of deep learning and artificial intelligence technologies, ultra-large-scale neural network models are widely used in fields such as natural language processing. These models have a huge number of parameters and high inference computation overhead, which limits their practical application in industrial scenarios.

[0003] The sparse architecture of Mixture-of-Experts (MoE) has become a typical approach to model expansion in recent years. By introducing gating functions and "expert" modules, it activates only a subset of experts in each inference iteration, thus achieving sublinear growth in computational cost with model capacity and enabling rapid expansion of model parameters. A model that activates k experts at a time is called a k-expert module. This architecture significantly improves the model's parameter scale and representational power while reducing the computational cost per sample. Sparse MoE structures are widely used in large models, such as the Google Switch Transformer and DeepSeek, and have become one of the core technologies driving the development of large-scale artificial intelligence (AI) models.

[0004] For large sparse MoE models, both input and output are sequences of text. During model inference, the gating function outputs the score of each expert and activates the k experts with the highest scores. Such a hybrid expert layer often occupies more than 90% of the memory of a large model.

[0005] In practical deployment and inference, the application of large-scale MoE models with a large number of parameters is often limited due to the limited memory resources of the Graphics Processing Unit (GPU). In this case, some expert modules need to be offloaded to secondary storage spaces, such as Central Processing Unit (CPU) memory or Solid State Drive (SSD), and data transfer with GPU memory is conducted via the Peripheral Component Interconnect Express (PCIe) bus. If an expert not on the GPU is needed during inference, that expert needs to be loaded onto the GPU for computation, replacing one expert originally on the GPU. This inference scenario is called offloading inference for large models. Therefore, in offloading inference scenarios with limited GPU memory, the selection of experts must not only consider which experts to activate to ensure the accuracy of the model output results, but also the location of the experts and the data transfer overhead between different storage media. Traditional expert inference methods often focus only on the quality of model output results, loading the k experts with the highest global scores onto the GPU for computation on demand; or, in pursuit of optimization of inference latency, reducing the number of experts used, or using only GPU experts in a single inference, sacrificing a significant amount of output quality and ignoring the need for coordinated optimization between expert distribution and data transmission.

[0006] Most existing technologies employ on-demand expert loading, incurring significant time delays for expert transfer; or they are limited to 2-expert models, focusing on improvements in expert module allocation. On-demand allocation is simple and easy to implement, but it doesn't consider the differences in inference latency between different experts, such as varying hardware resources and computational loads, which could lead to numerous expert transfer requests and extend overall inference time. Existing improvements have strong model limitations, such as being limited to 2-expert models, lacking scalability, and sacrificing considerable result quality. These two methods either prioritize result quality or latency / energy consumption, failing to achieve an overall balance between the two.

[0007] In the existing technology, there is a lack of a large model inference optimization method that balances model inference speed and output quality for large MoE models. Summary of the Invention

[0008] To address the technical problem in existing technologies where large MoE models in GPU-memory-constrained scenarios cannot achieve optimal synergy between model output quality and inference latency, this invention provides a method and apparatus for optimizing MoE large model inference based on dynamic scheduling of expert modules. The technical solution is as follows:

[0009] On the one hand, a method for MoE large model inference optimization based on dynamic scheduling of expert modules is provided. This method is implemented by a MoE large model inference optimization device and includes:

[0010] Upload the MoE large model to the computer, read the current hardware resource status, and obtain the GPU expert index list, CPU expert index list, and GPU candidate expert number value set.

[0011] Obtain the dialogue text dataset; based on the GPU expert index list and the set of GPU candidate expert number values, perform pre-inference using the MoE large model according to the dialogue text dataset to obtain the first evaluation score set and the first expert index list;

[0012] Based on the GPU expert index list and the first expert index list, the value set of the number of GPU candidate experts is optimized by the expert cache hit rate calculation method to obtain the second expert index list, the third expert index list and the optimized value set of GPU candidate experts.

[0013] Based on the relationship curve between the increase in expert scores and the number of experts transmitted, the CPU expert module is dynamically scheduled according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list to obtain the set of GPU unloaded experts and the CPU expert scheduling index list.

[0014] Based on the set of unloaded experts on the GPU, the GPU expert module is dynamically supplemented according to the first evaluation score set, the third expert index list and the CPU expert scheduling index list to obtain the GPU expert supplementary index list.

[0015] Based on the second expert index list, the CPU expert scheduling index list, and the GPU expert supplementary index list, a scheduled GPU expert index list is obtained. Based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, the MoE large model is used to perform re-inference based on the dialogue text dataset to obtain the inference result text dataset.

[0016] On the other hand, a MoE large model inference optimization device based on expert module dynamic scheduling is provided. This device is applied to the MoE large model inference optimization method based on expert module dynamic scheduling. The device includes:

[0017] The hardware information reading module is used to upload the MoE large model to the computer, read the current hardware resource status, and obtain the GPU expert index list, CPU expert index list, and GPU candidate expert number value set.

[0018] The first dialogue reasoning module is used to obtain the dialogue text dataset; based on the GPU expert index list and the set of GPU candidate expert number values, it performs pre-inference through the MoE large model according to the dialogue text dataset to obtain the first evaluation score set and the first expert index list.

[0019] The expert value optimization module is used to optimize the value set of the number of GPU candidate experts based on the GPU expert index list and the first expert index list by using the expert cache hit rate calculation method to obtain the second expert index list, the third expert index list and the optimized value set of GPU candidate experts.

[0020] The CPU expert scheduling module is used to dynamically schedule the CPU expert module based on the relationship curve between the increase value of expert scores and the number of experts transmitted, according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list, to obtain the set of unloaded experts on the GPU and the CPU expert scheduling index list.

[0021] The GPU expert supplementation module is used to dynamically supplement the GPU expert module based on the number of experts not loaded on the GPU, according to the first evaluation score set, the third expert index list and the CPU expert scheduling index list, and to obtain the GPU expert supplementation index list.

[0022] The second dialogue inference module is used to obtain the scheduled GPU expert index list based on the second expert index list, the CPU expert scheduling index list, and the GPU expert supplementary index list; based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, it performs re-inference through the MoE large model according to the dialogue text dataset to obtain the inference result text dataset.

[0023] On the other hand, a MoE large model inference optimization device is provided, the MoE large model inference optimization device comprising: a processor; a memory, the memory storing computer-readable instructions, which, when executed by the processor, implement any of the methods in the above-described MoE large model inference optimization method based on expert module dynamic scheduling.

[0024] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the methods in the above-described MoE large model inference optimization method based on expert module dynamic scheduling.

[0025] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0026] This invention proposes a MoE large-scale model inference optimization method based on dynamic scheduling of expert modules. By fine-grained planning of the number of expert modules loaded at each layer, it effectively improves cache hit rate. A threshold control mechanism is constructed based on the slope of the "expert score increase - expert transmission count" curve to achieve adaptive adjustment of experts. This invention can flexibly adjust the balance between inference speed and output quality according to specific application requirements, possessing strong customizability and versatility. While ensuring the overall output quality of the model, this invention dynamically schedules the activated expert sets at each layer by balancing the inference latency and expert score contribution at each layer, achieving a synergistic optimization of model output quality and inference latency. This invention is a large-scale model inference optimization method for MoE large-scale models that balances model inference speed and output quality. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a MoE large model inference optimization method based on dynamic scheduling of expert modules provided by an embodiment of the present invention;

[0029] Figure 2 This is a block diagram of a MoE large model inference optimization device based on dynamic scheduling of expert modules provided in an embodiment of the present invention;

[0030] Figure 3 This is a schematic diagram of the structure of a MoE large model inference optimization device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0032] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0033] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0034] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0035] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0036] This invention provides a MoE large model inference optimization method based on dynamic scheduling of expert modules. This method can be implemented by a MoE large model inference optimization device, which can be a terminal or a server. Figure 1 The flowchart shown is for the MoE large model inference optimization method based on dynamic scheduling of expert modules. The processing flow of this method may include the following steps:

[0037] S1. Upload the MoE large model to the computer, read the current hardware resource status, and obtain the GPU expert index list, CPU expert index list, and GPU candidate expert quantity value set.

[0038] In one feasible implementation, the MoE large language model in this invention consists of multiple decoding neural network layers, where L is the total number of layers. The model is divided into expert parameters and non-expert parameters. The non-expert parameter part is entirely loaded onto the GPU.

[0039] Each feedforward neural network layer consists of a gating function and several expert modules. An offloading mechanism is employed, loading some expert module parameters into GPU memory while storing the remaining parameters in host memory, reducing peak GPU memory load. The system reads the index list of expert modules currently loaded in the GPU and CPU, as well as the number of expert modules the GPU allows each decoding neural network layer to use, i.e., the number of candidate experts.

[0040] S2. Obtain the dialogue text dataset; based on the GPU expert index list and the set of GPU candidate expert quantity values, perform pre-inference using the MoE large model according to the dialogue text dataset to obtain the first evaluation score set and the first expert index list.

[0041] The MoE large model includes multiple decoding neural network layers;

[0042] The decoding neural network layer includes an attention layer, a feedforward neural network layer, and a normalization layer;

[0043] A feedforward neural network layer consists of a gating function and multiple expert modules; the multiple expert modules are multiple neural network modules with the same structure but different parameters.

[0044] In one feasible implementation, the MoE large model adopts a Transformer architecture, consisting of multiple decoding neural network layers. Each decoding neural network layer contains an attention layer, a feedforward neural network layer, and a normalization layer. The feedforward neural network layer of the MoE model consists of a gating function and several expert modules, also known as a hybrid expert layer.

[0045] Optionally, based on the GPU expert index list and the set of GPU candidate expert count values, and according to the dialogue text dataset, pre-inference is performed using the MoE large model to obtain the first evaluation score set and the first expert index list, including:

[0046] Based on the set of GPU candidate expert quantity values ​​and the GPU expert index list, a set of dialogue hidden state vectors is obtained by extracting hidden state vectors per token based on the dialogue text dataset.

[0047] Based on the set of dialogue hidden state vectors, the expert module evaluates the first set of evaluation scores by gating functions of multiple feedforward neural network layers.

[0048] Based on the descending order of the first evaluation score set, select the top k expert modules to generate the first expert index list.

[0049] In one feasible implementation, based on the expert modules currently loaded on the GPU and the number of GPU candidate experts, the dialogue text to be inferred is input into the MoE large model for inference to obtain the result text obtained from the dialogue text inference.

[0050] Specifically, a gating function is used to evaluate the expert modules in the corresponding decoding neural network layer, obtaining the first set of evaluation scores for all expert modules in the MoE large model. The evaluation scores are then sorted in descending order, and the top k expert modules are selected as the expert modules used for this inference, thus obtaining the first expert index list.

[0051] S3. Based on the GPU expert index list and the first expert index list, optimize the value set of the number of GPU candidate experts by using the expert cache hit rate calculation method to obtain the second expert index list, the third expert index list, and the optimized value set of GPU candidate experts.

[0052] Optionally, based on the GPU expert index list and the first expert index list, the preset set of values ​​for the number of GPU candidate experts is optimized using an expert cache hit rate calculation method to obtain a second expert index list, a third expert index list, and an optimized set of values ​​for GPU candidate experts, including:

[0053] Based on the first expert index list, the second expert index list that has been loaded on the GPU and used in the inference process and the third expert index list that has been loaded on the GPU and not used in the inference process are obtained by comparison and filtering according to the GPU expert index list.

[0054] The GPU's expert cache hit rate set is obtained by calculating based on the first expert index list and the second expert index list;

[0055] Based on the GPU expert index list, obtain the number of experts loaded on the GPU;

[0056] The GPU candidate expert optimization value set is obtained by calculating the expert cache hit rate set and the number of experts loaded on the GPU.

[0057] In one feasible implementation, the number of expert modules available for each decoding neural network layer is adaptively optimized in this step. During statistical inference, the number of times an expert parameter that is actually used already exists in GPU memory, i.e., the cache hit rate, represents the percentage of the total number of times that expert is invoked.

[0058] Based on the second expert index list loaded on the GPU and used during inference, and the third expert index list loaded on the GPU but not used during inference, the hit count of the expert module is obtained. For a certain decoding neural network layer i, the cache hit rate is the ratio of the hit count to the total number of calls preset by the large model.

[0059] By adaptively adjusting the number of experts loaded onto the GPU for each layer, layers with more evenly distributed expert usage can store more experts in GPU memory, while layers with more concentrated expert usage only require storing a smaller number of experts on the GPU. This improves the overall cache hit rate, increasing speed while better utilizing GPU resources.

[0060] The calculation method for the GPU candidate expert optimization value set is as follows (1):

[0061] (1);

[0062] Where L is the total number of decoding neural network layers in the MoE large model; C is the number of experts loaded on the GPU; For the optimized values ​​of the expert module in the i-th decoding neural network layer; Let be the expert cache hit rate of the i-th decoding neural network layer.

[0063] In one feasible implementation, the expert cache hit rate is defined as the proportion of experts used on the GPU. Based on the hit rate, the optimal values ​​for each layer of candidate experts are planned according to Equation 1 above.

[0064] S4. Based on the relationship curve between the increase in expert scores and the number of experts transmitted, dynamically schedule the CPU expert module according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list to obtain the set of unloaded experts on the GPU and the CPU expert scheduling index list.

[0065] Optionally, based on the relationship curve between the increase in expert scores and the number of experts transmitted, dynamic scheduling of the CPU expert module is performed according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list to obtain the set of GPU unloaded experts and the CPU expert scheduling index list, including:

[0066] The difference set is calculated based on the first expert index list and the second expert index list to obtain the set of experts not loaded by the GPU; the set of experts not loaded by the GPU is then used to determine the threshold set of candidate expert values.

[0067] Based on the threshold set of candidate experts, and according to the dialogue text dataset, the MoE large model is used to perform transmission granularity scanning to obtain the second evaluation score set under different numbers of candidate experts.

[0068] The set of expert score increases is calculated based on the second set of evaluation scores.

[0069] Data analysis was performed based on the set of expert score increases, and a set of curves showing the relationship between the increase in expert scores and the number of experts involved were plotted.

[0070] Based on a preset threshold for the number of transmission experts, the candidate expert set corresponding to the maximum slope is selected as the transmission expert set according to the set of relationship curves.

[0071] In the CPU expert index list, based on the first evaluation score set, select the number of expert modules corresponding to the number of transmission experts to obtain the CPU expert scheduling index list.

[0072] In one feasible implementation, after the text is input into the model, the gating layer of each MoE layer outputs the score of each expert and initially selects the top k experts for calculation. If r (r ≤ k) of these k experts are not on the GPU, then according to the selection strategy of this invention, p (p ≤ r) experts need to be transferred from host memory to the GPU and scheduled to the GPU using the PCIe bus.

[0073] During inference, for each MoE layer, the number of experts transmitted is p = 1, 2, …, r. The increase in GPU expert scores after expert transmission is collected, and a relationship curve is established between "expert score increase - number of transmitted experts". Since high-scoring experts are prioritized for transmission, the slope of this curve decreases. When selecting experts from host memory, selection proceeds sequentially from high-scoring experts downwards, and then a pre-set threshold for candidate expert values ​​is applied. Determine the number of experts that need to be transmitted. The process is as follows (2):

[0074] (2);

[0075] in, The increase in GPU expert scores obtained by transmitting p experts.

[0076] Candidate expert threshold The setting is optimal, either provided by the calibration dataset or user-defined, to adjust the preference between speed and result quality. To determine the optimal threshold... An end-to-end analysis of the relationship between result quality and inference latency was performed on a calibration dataset, with different thresholds set for each layer. The optimal threshold for each layer was determined without compromising result quality. .

[0077] S5. Based on the set of unloaded experts on the GPU, dynamically supplement the GPU expert module according to the first evaluation score set, the third expert index list, and the CPU expert scheduling index list to obtain the GPU expert supplementary index list.

[0078] Optionally, based on the set of unloaded experts on the GPU, the GPU expert module is dynamically supplemented according to the first evaluation score set, the third expert index list, and the CPU expert scheduling index list to obtain a GPU expert supplementary index list, including:

[0079] Based on the CPU expert scheduling index list, obtain the CPU expert scheduling quantity set;

[0080] The GPU expert supplementary set is calculated based on the GPU unloaded expert set and the CPU expert scheduling set;

[0081] In the third expert index list, based on the first evaluation score set, select the corresponding number of expert modules from the GPU expert supplementary quantity set to obtain the GPU expert supplementary index list.

[0082] In one feasible implementation, for the r–p experts that were not transmitted, the r–p experts with the highest scores who were not used in this inference are selected from the GPU to replace them. Transmitting suitable experts from host memory to the GPU can improve the quality of the output results, while controlling the number of experts transmitted and limiting the transmission time.

[0083] S6. Based on the second expert index list, the CPU expert scheduling index list, and the GPU expert supplementary index list, obtain the scheduled GPU expert index list; based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, and based on the dialogue text dataset, perform re-inference through the MoE large model to obtain the inference result text dataset.

[0084] In one feasible implementation, based on the optimized values ​​of GPU candidate experts, the expert modules corresponding to the selected scheduled GPU expert index list are scheduled to the GPU for computational inference. This ensures that only the more important experts are transmitted to the GPU, effectively solving the problem of frequent expert transmission and reduced result quality in large MoE models processing text dialogue tasks when GPU memory is limited.

[0085] This invention supports a trade-off between speed and result quality; the inference tendency of the model can be adjusted by changing the threshold, which can favor the output result quality or inference speed of the MoE large model.

[0086] This invention proposes a MoE large-scale model inference optimization method based on dynamic scheduling of expert modules. By fine-grained planning of the number of expert modules loaded at each layer, it effectively improves cache hit rate. A threshold control mechanism is constructed based on the slope of the "expert score increase - expert transmission count" curve to achieve adaptive adjustment of experts. This invention can flexibly adjust the balance between inference speed and output quality according to specific application requirements, possessing strong customizability and versatility. While ensuring the overall output quality of the model, this invention dynamically schedules the activated expert sets at each layer by balancing the inference latency and expert score contribution at each layer, achieving a synergistic optimization of model output quality and inference latency. This invention is a large-scale model inference optimization method for MoE large-scale models that balances model inference speed and output quality.

[0087] Figure 2This is a block diagram of a MoE large model inference optimization device based on dynamic scheduling of expert modules, provided in an embodiment of the present invention. This device is used for the MoE large model inference optimization method based on dynamic scheduling of expert modules. (Refer to...) Figure 2 The device includes a hardware information reading module 210, a first dialogue reasoning module 220, an expert value optimization module 230, a CPU expert scheduling module 240, a GPU expert supplementation module 250, and a second dialogue reasoning module 260. Among them:

[0088] The hardware information reading module 210 is used to upload the MoE large model to the computer, read the current hardware resource status, and obtain the GPU expert index list, CPU expert index list and GPU candidate expert quantity value set.

[0089] The first dialogue reasoning module 220 is used to obtain the dialogue text dataset; based on the GPU expert index list and the GPU candidate expert number value set, it performs pre-inference through the MoE large model according to the dialogue text dataset to obtain the first evaluation score set and the first expert index list.

[0090] The expert value optimization module 230 is used to optimize the value set of the number of GPU candidate experts by using the expert cache hit rate calculation method based on the GPU expert index list and the first expert index list, so as to obtain the second expert index list, the third expert index list and the optimized value set of GPU candidate experts.

[0091] CPU expert scheduling module 240 is used to dynamically schedule the CPU expert module based on the relationship curve between the increase value of expert scores and the number of transmitted experts, according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list and the GPU expert index list, to obtain the set of GPU unloaded experts and the CPU expert scheduling index list.

[0092] GPU expert supplementation module 250 is used to dynamically supplement the GPU expert module based on the number of experts not loaded on the GPU, according to the first evaluation score set, the third expert index list and the CPU expert scheduling index list, to obtain the GPU expert supplementation index list.

[0093] The second dialogue inference module 260 is used to obtain a scheduled GPU expert index list based on the second expert index list, the CPU expert scheduling index list, and the GPU expert supplementary index list; based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, and according to the dialogue text dataset, it performs re-inference through the MoE large model to obtain the inference result text dataset.

[0094] The MoE large model includes multiple decoding neural network layers;

[0095] The decoding neural network layer includes an attention layer, a feedforward neural network layer, and a normalization layer;

[0096] A feedforward neural network layer consists of a gating function and multiple expert modules; the multiple expert modules are multiple neural network modules with the same structure but different parameters.

[0097] Optionally, the first dialogue reasoning module 220 is further used for:

[0098] Based on the set of GPU candidate expert quantity values ​​and the GPU expert index list, a set of dialogue hidden state vectors is obtained by extracting hidden state vectors per token based on the dialogue text dataset.

[0099] Based on the set of dialogue hidden state vectors, the expert module evaluates the first set of evaluation scores by gating functions of multiple feedforward neural network layers.

[0100] Based on the descending order of the first evaluation score set, select the top k expert modules to generate the first expert index list.

[0101] Optionally, the expert value optimization module 230 is further used for:

[0102] Based on the first expert index list, the second expert index list that has been loaded on the GPU and used in the inference process and the third expert index list that has been loaded on the GPU and not used in the inference process are obtained by comparison and filtering according to the GPU expert index list.

[0103] The GPU's expert cache hit rate set is obtained by calculating based on the first expert index list and the second expert index list;

[0104] Based on the GPU expert index list, obtain the number of experts loaded on the GPU;

[0105] The GPU candidate expert optimization value set is obtained by calculating the expert cache hit rate set and the number of experts loaded on the GPU.

[0106] The calculation method for the GPU candidate expert optimization value set is as follows (1):

[0107] (1);

[0108] Where L is the total number of decoding neural network layers in the MoE large model; C is the number of experts loaded on the GPU; For the optimized values ​​of the expert module in the i-th decoding neural network layer; Let be the expert cache hit rate of the i-th decoding neural network layer.

[0109] Optionally, the CPU expert scheduling module 240 is further used for:

[0110] The difference set is calculated based on the first expert index list and the second expert index list to obtain the set of experts not loaded by the GPU; the set of experts not loaded by the GPU is then used to determine the threshold set of candidate expert values.

[0111] Based on the threshold set of candidate experts, and according to the dialogue text dataset, the MoE large model is used to perform transmission granularity scanning to obtain the second evaluation score set under different numbers of candidate experts.

[0112] The set of expert score increases is calculated based on the second set of evaluation scores.

[0113] Data analysis was performed based on the set of expert score increases, and a set of curves showing the relationship between the increase in expert scores and the number of experts involved were plotted.

[0114] Based on a preset threshold for the number of transmission experts, the candidate expert set corresponding to the maximum slope is selected as the transmission expert set according to the set of relationship curves.

[0115] In the CPU expert index list, based on the first evaluation score set, select the number of expert modules corresponding to the number of transmission experts to obtain the CPU expert scheduling index list.

[0116] Optionally, the GPU expert supplementary module 250 is further used for:

[0117] Based on the CPU expert scheduling index list, obtain the CPU expert scheduling quantity set;

[0118] The GPU expert supplementary set is calculated based on the GPU unloaded expert set and the CPU expert scheduling set;

[0119] In the third expert index list, based on the first evaluation score set, select the corresponding number of expert modules from the GPU expert supplementary quantity set to obtain the GPU expert supplementary index list.

[0120] This invention proposes a MoE large-scale model inference optimization method based on dynamic scheduling of expert modules. By fine-grained planning of the number of expert modules loaded at each layer, it effectively improves cache hit rate. A threshold control mechanism is constructed based on the slope of the "expert score increase - expert transmission count" curve to achieve adaptive adjustment of experts. This invention can flexibly adjust the balance between inference speed and output quality according to specific application requirements, possessing strong customizability and versatility. While ensuring the overall output quality of the model, this invention dynamically schedules the activated expert sets at each layer by balancing the inference latency and expert score contribution at each layer, achieving a synergistic optimization of model output quality and inference latency. This invention is a large-scale model inference optimization method for MoE large-scale models that balances model inference speed and output quality.

[0121] Figure 3 This is a schematic diagram of the structure of a MoE large model inference optimization device provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the MoE large model inference optimization apparatus may include the above-mentioned Figure 2 The illustrated MoE large model inference optimization device is based on dynamic scheduling of expert modules. Optionally, the MoE large model inference optimization device 310 may include a first processor 2001.

[0122] Optionally, the MoE large model inference optimization device 310 may also include a memory 2002 and a transceiver 2003.

[0123] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0124] The following is combined Figure 3 A detailed introduction to each component of the MoE large model inference optimization apparatus 310:

[0125] The first processor 2001 is the control center of the MoE large model inference optimization device 310. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0126] Optionally, the first processor 2001 can perform various functions of the MoE large model inference optimization device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0127] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0128] In a specific implementation, as one example, the MoE large model inference optimization device 310 may also include multiple processors, for example... Figure 3 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0129] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0130] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be accessed through the interface circuit of the MoE large model inference optimization device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0131] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0132] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0133] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the MoE large model inference optimization device 310. Figure 3 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0134] It should be noted that, Figure 3 The structure of the MoE large model inference optimization device 310 shown in the figure does not constitute a limitation on the router. The actual MoE large model inference optimization device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0135] Furthermore, the technical effect of the MoE large model inference optimization device 310 can be referred to the technical effect of the MoE large model inference optimization method based on expert module dynamic scheduling described in the above method embodiments, and will not be repeated here.

[0136] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0137] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0138] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0139] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0140] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0141] It should be understood that, in various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0142] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0143] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0144] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0145] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0147] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for MoE large model inference optimization based on dynamic scheduling of expert modules, characterized in that, The method includes: Upload the MoE large model to the computer, read the current hardware resource status, and obtain the GPU expert index list, CPU expert index list, and GPU candidate expert number value set. Obtain the dialogue text dataset; based on the GPU expert index list and the set of GPU candidate expert number values, perform pre-inference using the MoE large model according to the dialogue text dataset to obtain the first evaluation score set and the first expert index list; Based on the GPU expert index list and the first expert index list, the set of values ​​for the number of GPU candidate experts is optimized by the expert cache hit rate calculation method to obtain the second expert index list, the third expert index list and the optimized set of values ​​for GPU candidate experts. Based on the relationship curve between the increase in expert scores and the number of experts transmitted, the CPU expert module is dynamically scheduled according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list to obtain the set of GPU unloaded experts and the CPU expert scheduling index list. Based on the set of unloaded experts on the GPU, the GPU expert module is dynamically supplemented according to the first evaluation score set, the third expert index list and the CPU expert scheduling index list to obtain the GPU expert supplementary index list. Based on the second expert index list, the CPU expert scheduling index list, and the GPU expert supplementary index list, a scheduled GPU expert index list is obtained. Based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, and based on the dialogue text dataset, the MoE large model is used for re-inference to obtain the inference result text dataset.

2. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, The MoE large model includes multiple decoding neural network layers; The decoding neural network layer includes an attention layer, a feedforward neural network layer, and a normalization layer; The feedforward neural network layer includes a gating function and multiple expert modules; the multiple expert modules are multiple neural network modules with the same structure but different parameters.

3. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 2, characterized in that, The method, based on the GPU expert index list and the set of GPU candidate expert count values, uses the MoE large model for pre-inference based on the dialogue text dataset to obtain the first evaluation score set and the first expert index list, including: Based on the set of GPU candidate expert quantity values ​​and the GPU expert index list, a set of dialogue hidden state vectors is obtained by extracting hidden state vectors per token based on the dialogue text dataset. Based on the set of dialogue hidden state vectors, the expert module evaluates the first set of evaluation scores by gating functions of multiple feedforward neural network layers. Based on the descending order of the first evaluation score set, select the top k expert modules to generate the first expert index list.

4. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, The step of optimizing the preset set of values ​​for the number of GPU candidate experts based on the GPU expert index list and the first expert index list using an expert cache hit rate calculation method to obtain the second expert index list, the third expert index list, and the optimized set of values ​​for GPU candidate experts includes: Based on the first expert index list, the second expert index list that has been loaded on the GPU and used in the inference process and the third expert index list that has been loaded on the GPU and not used in the inference process are obtained by comparison and filtering according to the GPU expert index list. The GPU's expert cache hit rate set is obtained by calculating based on the first expert index list and the second expert index list; Based on the GPU expert index list, obtain the number of experts loaded on the GPU; The GPU candidate expert optimization value set is obtained by calculating the expert cache hit rate set and the number of experts loaded on the GPU.

5. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 4, characterized in that, The calculation method for the GPU candidate expert optimization value set is as follows: (1) （1）； Where L is the total number of decoding neural network layers in the MoE large model; C is the number of experts loaded on the GPU; Let be the optimized value of the expert module in the i-th decoding neural network layer; Let be the expert cache hit rate of the i-th decoding neural network layer.

6. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, The relationship curve between the increase in expert scores and the number of transmitted experts is used to dynamically schedule the CPU expert module according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list. This results in the acquisition of the set of GPU-unloaded experts and the CPU expert scheduling index list, including: The difference set is calculated based on the first expert index list and the second expert index list to obtain the set of experts not loaded by the GPU; the set of experts not loaded by the GPU is then used to determine the threshold set of candidate expert values. Based on the threshold set of candidate experts, and according to the dialogue text dataset, the MoE large model is used to perform transmission granularity scanning to obtain the second evaluation score set under different numbers of candidate experts. The set of expert score increases is calculated based on the second set of evaluation scores. Data analysis was performed based on the set of expert score increases, and a set of curves showing the relationship between the increase in expert scores and the number of experts involved were plotted. Based on a preset threshold for the number of transmission experts, the candidate expert set corresponding to the maximum slope is selected as the transmission expert set according to the set of relationship curves. In the CPU expert index list, based on the first evaluation score set, select the number of expert modules corresponding to the number of transmission experts to obtain the CPU expert scheduling index list.

7. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, The process involves dynamically supplementing the GPU expert module based on the set of unloaded experts on the GPU, according to the first evaluation score set, the third expert index list, and the CPU expert scheduling index list, to obtain a GPU expert supplementary index list, including: Based on the CPU expert scheduling index list, obtain the CPU expert scheduling quantity set; The GPU expert supplementary set is calculated based on the GPU unloaded expert set and the CPU expert scheduling set; In the third expert index list, based on the first evaluation score set, select the corresponding number of expert modules from the GPU expert supplementary quantity set to obtain the GPU expert supplementary index list.

8. A MoE large model inference optimization device based on expert module dynamic scheduling, wherein the MoE large model inference optimization device based on expert module dynamic scheduling is used to implement the MoE large model inference optimization method based on expert module dynamic scheduling as described in any one of claims 1-7, characterized in that, The device includes: The hardware information reading module is used to upload the MoE large model to the computer, read the current hardware resource status, and obtain the GPU expert index list, CPU expert index list, and GPU candidate expert number value set. The first dialogue reasoning module is used to obtain the dialogue text dataset; based on the GPU expert index list and the set of GPU candidate expert number values, it performs pre-inference through the MoE large model according to the dialogue text dataset to obtain the first evaluation score set and the first expert index list. The expert value optimization module is used to optimize the value set of the number of GPU candidate experts based on the GPU expert index list and the first expert index list by using the expert cache hit rate calculation method to obtain the second expert index list, the third expert index list and the optimized value set of GPU candidate experts. The CPU expert scheduling module is used to dynamically schedule the CPU expert module based on the relationship curve between the increase value of expert scores and the number of experts transmitted, according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list, to obtain the set of unloaded experts on the GPU and the CPU expert scheduling index list. The GPU expert supplementation module is used to dynamically supplement the GPU expert module based on the number of experts not loaded on the GPU, according to the first evaluation score set, the third expert index list and the CPU expert scheduling index list, and to obtain the GPU expert supplementation index list. The second dialogue inference module is used to obtain the scheduled GPU expert index list based on the second expert index list, the CPU expert scheduling index list, and the GPU expert supplementary index list; based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, it performs re-inference through the MoE large model according to the dialogue text dataset to obtain the inference result text dataset.

9. A MoE large model inference optimization device, characterized in that, The MoE large model inference optimization device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images