MoE large model inference optimization method and device based on dynamic scheduling of expert module

Abstract

The application discloses a MoE large model inference optimization method and device based on expert module dynamic scheduling, and relates to the technical field of large model inference optimization. The method comprises the following steps: performing pre-inference on dialogue text data sets through a MoE large model to obtain a first evaluation score set; performing value optimization on a GPU candidate expert quantity value set through an expert cache hit rate calculation method to obtain a GPU candidate expert optimized value set; performing CPU expert module dynamic scheduling based on a relationship curve of expert score rising value-transferred expert quantity to obtain a CPU expert scheduling index list; performing GPU expert module dynamic supplementing according to the first evaluation score set and the CPU expert scheduling index list to obtain a GPU expert supplementing index list, and performing re-inference through the MoE large model to obtain an inference result text data set. The application is a large model inference optimization method for balancing the model inference speed and the output result quality of the MoE large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model inference optimization, in particular to a MoE large model inference optimization method and device based on expert module dynamic scheduling. BACKGROUND

[0002] With the continuous development of deep learning and artificial intelligence technology, super large scale neural network models are widely used in natural language processing and other fields. These models have huge parameters and high inference calculation overhead, which limits their application in practical industrial scenarios.

[0003] The sparse architecture of the mixture of experts (MoE) is a typical way of model expansion in recent years. By introducing a gating function and "expert" Expert module, only a part of the experts is activated for each inference, so that the computational overhead grows sublinearly with the model capacity, achieving the effect of rapid expansion of model parameters. For a model that activates k experts each time, it is called a k-expert module. Such architecture greatly improves the parameter size and representation ability of the model, while reducing the computational cost per sample. Sparse MoE structure is widely used in large models such as Google Switch Transformer, DeepSeek, etc., and has become one of the core technologies to promote the development of artificial intelligence (AI) large models.

[0004] For sparse MoE large models, the input and output are both sequences composed of text. When the model is inferred, the gating function will output the score of each expert, and the k experts with the highest score will be activated. Such a mixed expert layer often accounts for more than 90% of the memory of a large model.

[0005] In actual deployment and inference process, due to the limited memory resources of a graphics processing unit (GPU), the application of a MoE large model with an ultra-large scale of parameters is often limited. In this case, part of the expert modules need to be offloaded to a secondary storage space, such as a central processing unit (CPU) memory, a solid state drive (SSD), etc., and data transmission is performed with the GPU memory through a peripheral component interconnect express (PCIe) bus. If an expert not on the GPU is needed in the inference process, the expert needs to be loaded onto the GPU for calculation and replace an expert originally on the GPU. Such an inference scenario is referred to as offloaded inference of a large model. Therefore, in the offloaded inference scenario with limited GPU memory, the selection of an expert needs to consider not only which experts to activate to ensure the accuracy of the model output result, but also the location of the expert and the transmission overhead of data between different storage media. Traditional expert inference methods often only focus on the quality of the model output result, and load the k experts with the highest global scores to the GPU for calculation in an on-demand loading manner; or in order to optimize the inference delay, the number of experts used is reduced, or only GPU experts are used in one inference, which sacrifices a lot of output result quality and ignores the collaborative optimization demand between the expert distribution and data transmission.

[0006] Most of the prior art uses the on-demand loading method of experts, which spends a lot of time delay for expert transmission; or is limited to a 2-expert model and improves the allocation of expert modules. The on-demand allocation method is simple and easy to implement, but does not consider the inference delay differences between different experts, such as different hardware resources and different computing loads between experts, which may lead to a large number of expert transmission requirements and prolong the overall inference time. The existing improved method has strong model limitations, such as being limited to a 2-expert model, lacking scalability, and sacrificing a lot of result quality. These two methods either only pursue result quality or only pursue delay / energy consumption, and cannot achieve overall balance between the two.

[0007] In the prior art, there is a lack of a large model inference optimization method for balancing the model inference speed and the output result quality of a MoE large model. SUMMARY

[0008] To solve the technical problem in the prior art that a MoE large model cannot achieve collaborative optimization of model output quality and inference delay in a GPU memory limited scenario, an embodiment of the present application provides a MoE large model inference optimization method and device based on dynamic scheduling of expert modules. The technical solution is as follows:

[0009] In one aspect, a MoE large model inference optimization method based on expert module dynamic scheduling is provided. The method is implemented by a MoE large model inference optimization device. The method comprises the following steps:

[0010] uploading the MoE large model to a computer, reading the current hardware resource state, obtaining a GPU expert index list, a CPU expert index list, and a GPU candidate expert quantity value set;

[0011] obtaining a dialogue text dataset; based on the GPU expert index list and the GPU candidate expert quantity value set, performing pre-inference on the dialogue text dataset by the MoE large model to obtain a first evaluation score set and a first expert index list;

[0012] performing value optimization on the GPU candidate expert quantity value set by an expert cache hit rate calculation method according to the GPU expert index list and the first expert index list to obtain a second expert index list, a third expert index list, and a GPU candidate expert optimized value set;

[0013] based on a relationship curve of expert score rise value-transferred expert quantity, performing CPU expert module dynamic scheduling according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list to obtain a GPU unloaded expert quantity set and a CPU expert scheduling index list;

[0014] based on the GPU unloaded expert quantity set, performing GPU expert module dynamic supplement according to the first evaluation score set, the third expert index list, and the CPU expert scheduling index list to obtain a GPU expert supplement index list;

[0015] obtaining a scheduled GPU expert index list according to the second expert index list, the CPU expert scheduling index list, and the GPU expert supplement index list; based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, performing re-inference on the dialogue text dataset by the MoE large model to obtain an inference result text dataset.

[0016] In another aspect, a MoE large model inference optimization device based on expert module dynamic scheduling is provided. The device is applied to a MoE large model inference optimization method based on expert module dynamic scheduling. The device comprises the following modules:

[0017] a hardware information reading module, configured to upload the MoE large model to a computer, read the current hardware resource state, obtain a GPU expert index list, a CPU expert index list, and a GPU candidate expert quantity value set;

[0018] The first dialogue reasoning module is configured to obtain dialogue text data sets; based on a GPU expert index list and a GPU candidate expert quantity value set, first evaluation score sets and a first expert index list are obtained by pre-reasoning through a MoE large model according to the dialogue text data sets;

[0019] The expert value optimization module is configured to perform value optimization on the GPU candidate expert quantity value set through an expert cache hit rate calculation method according to the GPU expert index list and the first expert index list, and obtain a second expert index list, a third expert index list and a GPU candidate expert optimized value set;

[0020] The CPU expert scheduling module is configured to perform CPU expert module dynamic scheduling according to the first expert index list, the second expert index list, the first evaluation score sets, a CPU expert index list and a GPU expert index list based on a relationship curve of expert score rising value-transmission expert quantity, and obtain a GPU unloaded expert quantity set and a CPU expert scheduling index list.

[0021] The GPU expert supplement module is configured to perform GPU expert module dynamic supplement according to the first evaluation score sets, the third expert index list and the CPU expert scheduling index list based on the GPU unloaded expert quantity set, and obtain a GPU expert supplement index list.

[0022] The second dialogue reasoning module is configured to obtain a scheduled GPU expert index list according to the second expert index list, the CPU expert scheduling index list and the GPU expert supplement index list, and obtain a reasoning result text data set by re-reasoning through a MoE large model according to dialogue text data sets based on the scheduled GPU expert index list and the GPU candidate expert optimized value set.

[0023] In another aspect, a MoE large model reasoning optimization device is provided, which comprises a processor and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the above-mentioned MoE large model reasoning optimization methods based on expert module dynamic scheduling.

[0024] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the above-mentioned MoE large model reasoning optimization methods based on expert module dynamic scheduling.

[0025] The technical scheme provided by the embodiments of the present application has at least the following beneficial effects:

[0026] The application provides a MoE large model inference optimization method based on expert module dynamic scheduling, which effectively improves the cache hit rate through fine-grained planning of the number of loaded expert modules in each layer. A threshold control mechanism is constructed according to the slope of the "expert score rise value-expert transmission quantity" curve, realizing the adaptive adjustment of the experts. The application can flexibly adjust the balance point of inference speed and output quality according to specific application requirements, and has strong customizability and universality. On the basis of ensuring the overall output result quality of the model, the application dynamically schedules the activated expert set of each layer by weighing the inference delay and expert score contribution of each layer, realizing the collaborative optimization of model output quality and inference delay. The application is a large model inference optimization method for balancing the inference speed and output result quality of the MoE large model. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0028] Figure 1 is a MoE large model inference optimization method flowchart based on expert module dynamic scheduling provided by the embodiment of the present application;

[0029] Figure 2 is a MoE large model inference optimization device block diagram based on expert module dynamic scheduling provided by the embodiment of the present application;

[0030] Figure 3 is a structural schematic diagram of a MoE large model inference optimization device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the present application will be described below with reference to the drawings.

[0032] In the embodiments of the present application, the words such as "example", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0033] In the embodiments of the present application, "image" and "picture" can be used interchangeably, and it should be pointed out that their meanings are consistent when their differences are not emphasized.

[0034] In the embodiments of the present application, sometimes the subscript such as W1 can be written in the form of non-subscript such as W1, and their meanings are consistent when their differences are not emphasized.

[0035] To make the technical problems, technical solutions and advantages to be solved by the present application clearer, the following will be described in detail in conjunction with the drawings and specific embodiments.

[0036] The embodiments of the present application provide a MoE large model inference optimization method based on expert module dynamic scheduling, which can be implemented by a MoE large model inference optimization device. The MoE large model inference optimization device can be a terminal or a server. As shown in the MoE large model inference optimization method flowchart based on expert module dynamic scheduling, the processing flow of the method can include the following steps: Figure 1

[0037] S1, upload the MoE large model to the computer, read the current hardware resource state, obtain the GPU expert index list, the CPU expert index list and the GPU candidate expert quantity value set.

[0038] In a feasible implementation manner, in the present application, the MoE large language model is composed of a plurality of decoding neural network layers, and L is the total number of layers. The model is divided into two parts of expert parameters and non-expert parameters. The non-expert parameter part is all loaded to the GPU.

[0039] For each feedforward neural network layer, it is composed of a gating function and a plurality of expert modules. An offloading mechanism is adopted, part of the expert module parameters are loaded to the GPU memory, and the rest of the expert module parameters are saved in the host memory, so as to reduce the peak pressure of the GPU memory. The index list of the loaded expert modules in the current GPU and CPU, and the number of expert modules that can be used by each decoding neural network layer in the GPU, i.e. the candidate expert quantity value, are read.

[0040] S2, obtain the dialogue text data set; based on the GPU expert index list and the GPU candidate expert quantity value set, according to the dialogue text data set, pre-inference is performed through the MoE large model to obtain a first evaluation score set and a first expert index list.

[0041] The MoE large model includes a plurality of decoding neural network layers.​

[0042] The decoding neural network layer includes an attention layer, a feedforward neural network layer, and a normalization layer.

[0043] The feedforward neural network layer includes a gating function and a plurality of expert modules; the plurality of expert modules are a plurality of neural network modules with the same structure and different parameters.

[0044] In a feasible implementation, the MoE large model adopts a Transformer architecture and is composed of a plurality of decoding neural network layers, each decoding neural network layer including an attention layer, a feedforward neural network layer, and a normalization layer. The feedforward neural network layer of the MoE model is composed of a gating function and a plurality of expert modules, also known as a mixed expert layer.

[0045] Optionally, based on the GPU expert index list and the GPU candidate expert quantity value set, a first evaluation score set and a first expert index list are obtained by pre-reasoning the dialogue text dataset through the MoE large model, including:

[0046] Based on the GPU candidate expert quantity value set and the GPU expert index list, a dialogue hidden state vector set is obtained by performing token-by-token hidden state vector extraction on the dialogue text dataset;

[0047] Based on the dialogue hidden state vector set, a first evaluation score set is obtained by evaluating the expert modules through the gating function of the plurality of feedforward neural network layers.

[0048] According to the descending order arrangement of the first evaluation score set, the first expert index list is generated by selecting the top k expert modules.

[0049] In a feasible implementation, based on the expert modules loaded by the current GPU and the GPU candidate expert quantity value, the dialogue text to be reasoned is input into the MoE large model for reasoning to obtain a result text reasoned by the dialogue text.

[0050] The gating function is used to evaluate the expert modules in the decoding neural network layer to obtain a first evaluation score set corresponding to all expert modules in the MoE large model. The evaluation scores are arranged in descending order, and the top k expert modules with high scores are selected as the expert modules used for this reasoning, thereby obtaining a first expert index list.

[0051] S3, according to the GPU expert index list and the first expert index list, the GPU candidate expert quantity value set is optimized by the expert cache hit rate calculation method to obtain a second expert index list, a third expert index list, and a GPU candidate expert optimized value set.

[0052] Optionally, according to the GPU expert index list and the first expert index list, the preset GPU candidate expert number value set is valued and optimized by the expert cache hit rate calculation method to obtain a second expert index list, a third expert index list and a GPU candidate expert optimized value set, including:

[0053] Based on the first expert index list, the GPU expert index list is compared and screened to obtain a second expert index list loaded on the GPU and used in the inference process and a third expert index list loaded on the GPU and not used in the inference process;

[0054] According to the first expert index list and the second expert index list, the expert cache hit rate set of the GPU is calculated;

[0055] According to the GPU expert index list, the number of loaded experts of the GPU is obtained;

[0056] According to the expert cache hit rate set and the number of loaded experts of the GPU, the GPU candidate expert optimized value set is calculated.

[0057] In a feasible implementation, the number of available expert modules for each decoding neural network layer is adaptively optimized and adjusted in this step. In the statistical inference process, the number of times that the expert parameters actually used exist in the GPU video memory, that is, the cache hit rate, represents the percentage of the total number of times that the expert is called.

[0058] Based on the second expert index list loaded on the GPU and used in the inference process and the third expert index list loaded on the GPU and not used in the inference process, the hit number of the expert module is obtained. For a decoding neural network layer i, the cache hit rate is the ratio of the hit number to the total number of calls preset by the large model.

[0059] After adaptively adjusting the number of experts loaded into the GPU for each layer, the layer with more average expert use can store more experts into the GPU memory, and the layer with more concentrated expert use only needs to store a small number of experts into the GPU. The overall cache hit rate is improved, which improves the speed while better utilizing the GPU resources.

[0060] The calculation method of the GPU candidate expert optimized value set is as follows formula (1):

[0061] (1);

[0062] Wherein, L is the total number of decoding neural network layers in the MoE large model; C is the number of loaded experts of the GPU; is the optimized value of the expert module in the i-th decoding neural network layer; The expert cache hit rate of the ith decoding neural network layer.

[0063] In an implementable embodiment, the expert cache hit rate is defined as the proportion of the used experts on the GPU. According to the hit rate, the optimization value of each layer of candidate experts is planned according to the above formula 1.

[0064] S4, based on the relationship curve of the expert score rise value-transmission expert quantity, the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list and the GPU expert index list are used for CPU expert module dynamic scheduling to obtain the GPU unloaded expert quantity set and the CPU expert scheduling index list.

[0065] Optionally, based on the relationship curve of the expert score rise value-transmission expert quantity, the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list and the GPU expert index list are used for CPU expert module dynamic scheduling to obtain the GPU unloaded expert quantity set and the CPU expert scheduling index list, comprising:

[0066] According to the first expert index list and the second expert index list, the difference set calculation is performed to obtain the GPU unloaded expert quantity set; and according to the GPU unloaded expert quantity set, the candidate expert value threshold set is determined;

[0067] Based on the candidate expert value threshold set, the transmission granularity scanning is performed on the dialogue text data set using the MoE large model to obtain the second evaluation score set under different candidate expert quantities;

[0068] According to the second evaluation score set, the expert score rise value set is obtained by calculation;

[0069] According to the expert score rise value set, data analysis is performed, and a relationship curve set of the expert score rise value-transmission expert quantity is drawn;

[0070] Based on the preset transmission expert quantity threshold, according to the relationship curve set, the candidate expert quantity set corresponding to the maximum slope is selected as the transmission expert quantity set;

[0071] In the CPU expert index list, according to the first evaluation score set, the expert module corresponding to the number of the transmission expert quantity set is selected to obtain the CPU expert scheduling index list.

[0072] In a feasible implementation, after inputting the text into the model, the gating layer outputs the scores of each expert and preliminarily selects the top k experts for calculation when passing through each MoE layer. If there are r (r ≤ k) experts in the k experts that are not on the GPU, the p (p ≤ r) experts that need to be transferred from the host memory to the GPU are determined by the selection strategy of the application, and are dispatched to the GPU using the PCIe bus.

[0073] In the inference process, for each MoE layer, the GPU expert score increase value caused by the transmission of experts is collected, and a "expert score increase value-transmitted expert quantity" relationship curve is established. Since high-score experts are preferentially transmitted, the slope of the curve decreases. When selecting experts from the host memory, high-score experts are selected in turn, and the candidate expert value threshold is preset to determine the number of experts to be transmitted , which is as follows (2):

[0074] (2);

[0075] wherein, is the GPU expert score increase value obtained by transmitting p experts.

[0076] The candidate expert value threshold is set by an optimal value given by a calibration data set or set by a user to adjust the tendency of speed and result quality. To determine the optimal threshold , an end-to-end analysis of the relationship between result quality and inference delay is performed on the calibration data set, and different thresholds are set for each layer. Without affecting the result quality, the best threshold for each layer is determined .

[0077] S5, based on the set of GPU unloaded experts, the first evaluation score set, the third expert index list and the CPU expert scheduling index list are used to dynamically supplement the GPU expert module to obtain a GPU expert supplement index list.

[0078] Optionally, based on the set of GPU unloaded experts, the first evaluation score set, the third expert index list and the CPU expert scheduling index list are used to dynamically supplement the GPU expert module to obtain a GPU expert supplement index list, including:

[0079] According to the CPU expert scheduling index list, a set of CPU expert scheduling quantities is obtained;

[0080] According to the set of GPU unloaded experts and the set of CPU expert scheduling quantities, a set of GPU expert supplement quantities is obtained;

[0081] In the third expert index list, according to the first evaluation score set, the corresponding number of expert modules of the GPU expert supplement number set is selected to obtain a GPU expert supplement index list.

[0082] In a feasible implementation, in the present application, for the r-p experts not transmitted, the top r-p experts not used in this inference are selected from the GPU to replace them. Transmitting the appropriate experts from the host memory to the GPU can improve the quality of the output results, while controlling the number of transmitted experts and limiting the time consumed by transmission.

[0083] S6, according to the second expert index list, the CPU expert scheduling index list and the GPU expert supplement index list, obtain a scheduled GPU expert index list; based on the scheduled GPU expert index list and the GPU candidate expert optimization value set, according to the dialogue text data set, re-inference is carried out through the MoE large model to obtain an inference result text data set.

[0084] In a feasible implementation, based on the GPU candidate expert optimization value, the selected expert modules corresponding to the scheduled GPU expert index list are scheduled to the GPU for participating in calculation and inference. In this way, only the more important part of the experts is transmitted to the GPU, effectively solving the problem of frequent transmission of experts and reduced result quality when the MoE large model processes text dialogue tasks while the GPU memory is limited.

[0085] The present application supports the trade-off between speed and result quality; the inference tendency of the model can be adjusted by changing the threshold value, and the output result quality or inference speed of the MoE large model can be inclined;

[0086] The present application proposes a MoE large model inference optimization method based on dynamic scheduling of expert modules, which effectively improves the cache hit rate through fine-grained planning of the number of expert modules loaded by each layer. A threshold control mechanism is constructed according to the slope of the "expert score rise value-expert transmission number" curve to realize the adaptive adjustment of experts. The present application can flexibly adjust the balance point of inference speed and output quality according to specific application requirements, and has strong customizability and universality. On the basis of ensuring the overall output result quality of the model, the present application dynamically schedules the activated expert set of each layer by balancing the inference delay of each layer and the contribution of the expert score, realizing the collaborative optimization of model output quality and inference delay. The present application is a large model inference optimization method for balancing the inference speed and output result quality of the MoE large model.

[0087] Figure 2The device is used for the MoE large model inference optimization method based on the expert module dynamic scheduling. Figure 2 The device comprises a hardware information reading module 210, a first dialogue inference module 220, an expert value optimization module 230, a CPU expert scheduling module 240, a GPU expert supplement module 250, and a second dialogue inference module 260.

[0088] The hardware information reading module 210 is used for uploading the MoE large model to a computer, reading the current hardware resource state, obtaining a GPU expert index list, a CPU expert index list, and a GPU candidate expert quantity value set;

[0089] The first dialogue inference module 220 is used for obtaining a dialogue text data set; based on the GPU expert index list and the GPU candidate expert quantity value set, a first evaluation score set and a first expert index list are obtained by pre-inference through the MoE large model according to the dialogue text data set;

[0090] The expert value optimization module 230 is used for performing value optimization on the GPU candidate expert quantity value set through an expert cache hit rate calculation method according to the GPU expert index list and the first expert index list, to obtain a second expert index list, a third expert index list, and a GPU candidate expert optimized value set;

[0091] The CPU expert scheduling module 240 is used for performing CPU expert module dynamic scheduling according to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list based on an expert score rise value-transmission expert quantity relationship curve, to obtain a GPU unloaded expert quantity set and a CPU expert scheduling index list;

[0092] The GPU expert supplement module 250 is used for performing GPU expert module dynamic supplement according to the first evaluation score set, the third expert index list, and the CPU expert scheduling index list based on the GPU unloaded expert quantity set, to obtain a GPU expert supplement index list;

[0093] The second dialogue inference module 260 is used for obtaining a scheduled GPU expert index list according to the second expert index list, the CPU expert scheduling index list, and the GPU expert supplement index list; and obtaining a reasoning result text data set by re-inference through the MoE large model according to the dialogue text data set based on the scheduled GPU expert index list and the GPU candidate expert optimized value set.

[0094] The MoE large model comprises a plurality of decoding neural network layers.

[0095] The decoding neural network layer includes an attention layer, a feedforward neural network layer, and a normalization layer.

[0096] The feedforward neural network layer includes one gating function and a plurality of expert modules; the plurality of expert modules are a plurality of neural network modules that are structurally identical and different in parameters.

[0097] Optionally, the first dialogue reasoning module 220 is further configured to:

[0098] Based on the GPU candidate expert value set and the GPU expert index list, a token hidden state vector set is extracted from the dialogue text data set to obtain a dialogue hidden state vector set;

[0099] According to the dialogue hidden state vector set, expert module evaluation is performed through the gating function of the plurality of feedforward neural network layers to obtain a first evaluation score set;

[0100] According to the descending order of the first evaluation score set, the first k expert modules are selected to generate a first expert index list.

[0101] Optionally, the expert value optimization module 230 is further configured to:

[0102] Based on the first expert index list, comparison and screening are performed according to the GPU expert index list to obtain a second expert index list that has been loaded on the GPU and used in the reasoning process, and a third expert index list that has been loaded on the GPU and not used in the reasoning process;

[0103] According to the first expert index list and the second expert index list, a GPU expert cache hit rate set is obtained;

[0104] According to the GPU expert index list, the number of loaded experts on the GPU is obtained;

[0105] According to the expert cache hit rate set and the number of loaded experts on the GPU, a GPU candidate expert optimization value set is obtained.

[0106] The calculation method of the GPU candidate expert optimization value set is as follows:

[0107] (1);

[0108] Wherein, L is the total number of decoding neural network layers in the MoE large model; C is the number of loaded experts on the GPU; is the optimization value of the expert module in the ith decoding neural network layer; is the expert cache hit rate of the ith decoding neural network layer.

[0109] Optionally, the CPU expert scheduling module 240 is further used for:

[0110] performing difference set calculation according to the first expert index list and the second expert index list to obtain a GPU unloaded expert quantity set; and determining a candidate expert value threshold set according to the GPU unloaded expert quantity set;

[0111] performing transmission granularity scanning on the dialogue text data set using the MoE large model based on the candidate expert value threshold set to obtain a second evaluation score set under different candidate expert quantities;

[0112] performing calculation according to the second evaluation score set to obtain an expert score rising value set;

[0113] performing data analysis according to the expert score rising value set and drawing a relationship curve set of expert score rising value-transmission expert quantity;

[0114] selecting a candidate expert quantity set corresponding to the maximum slope as the transmission expert quantity set according to the relationship curve set based on a preset transmission expert quantity threshold;

[0115] selecting expert modules corresponding in quantity to the transmission expert quantity set from the CPU expert index list according to the first evaluation score set to obtain a CPU expert scheduling index list.

[0116] Optionally, the GPU expert supplement module 250 is further used for:

[0117] obtaining a CPU expert scheduling quantity set according to the CPU expert scheduling index list;

[0118] performing calculation according to the GPU unloaded expert quantity set and the CPU expert scheduling quantity set to obtain a GPU expert supplement quantity set;

[0119] selecting expert modules corresponding in quantity to the GPU expert supplement quantity set from the third expert index list according to the first evaluation score set to obtain a GPU expert supplement index list.

[0120] The application provides a MoE large model inference optimization method based on expert module dynamic scheduling, which effectively improves the cache hit rate through fine-grained planning of the number of loaded expert modules in each layer. A threshold control mechanism is constructed according to the slope of the "expert score rise value-expert transmission quantity" curve to realize the adaptive adjustment of experts. The application can flexibly adjust the balance point of inference speed and output quality according to specific application requirements, and has strong customizability and universality. On the basis of ensuring the overall output result quality of the model, the application dynamically schedules the activated expert set of each layer by weighing the inference delay and expert score contribution of each layer, so as to realize the collaborative optimization of model output quality and inference delay. The application is a large model inference optimization method for balancing the inference speed and output result quality of the MoE large model.

[0121] Figure 3 is a structural schematic diagram of a MoE large model inference optimization device provided by an embodiment of the application, as shown in Figure 3 The MoE large model inference optimization device can include the MoE large model inference optimization apparatus based on expert module dynamic scheduling shown in Figure 2 Optionally, the MoE large model inference optimization device 310 can include a first processor 2001.

[0122] Optionally, the MoE large model inference optimization device 310 can further include a memory 2002 and a transceiver 2003.

[0123] The first processor 2001, the memory 2002 and the transceiver 2003 can be connected through a communication bus.

[0124] The various constituent components of the MoE large model inference optimization device 310 will be specifically introduced below: Figure 3 The first processor 2001 is the control center of the MoE large model inference optimization device 310, and can be one processor or a collective term of multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPU), can be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the application, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).

[0125]

[0126] ​Optionally, the first processor 2001 can perform various functions of the MoE large model inference optimization device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0127] In a specific implementation, as an example, the first processor 2001 can include one or more CPUs, such as the CPU0 and the CPU1 shown in FIG. 2. Figure 3

[0128] In a specific implementation, as an example, the MoE large model inference optimization device 310 can also include multiple processors, such as the first processor 2001 and the second processor 2004 shown in FIG. 2. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (for example, computer program instructions). Figure 3

[0129] The memory 2002 is configured to store software programs for implementing the solutions of the present application, and the first processor 2001 is configured to control the execution. For specific implementation manners, refer to the above-mentioned method embodiments, which will not be repeated here.

[0130] Optionally, the memory 2002 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including a compact disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory 2002 can be integrated with the first processor 2001 or exist independently and be coupled with the first processor 2001 through an interface circuit (not shown in FIG. 2) of the MoE large model inference optimization device 310, and the embodiments of the present application are not limited in this regard. Figure 3

[0131] ​​​The transceiver 2003 is configured to communicate with a network device or a terminal device.

[0132] Optionally, the transceiver 2003 can include a receiver and a transmitter (not shown in the figure). Figure 3 The receiver is configured to implement a receiving function, and the transmitter is configured to implement a transmitting function.

[0133] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently and be coupled to the first processor 2001 through an interface circuit (not shown in the figure) of the MoE large model inference optimization device 310. Figure 3 The embodiments of the present application do not make a specific limitation hereon.

[0134] It should be noted that the structure of the MoE large model inference optimization device 310 shown in the figure does not constitute a limitation on the router. Figure 3 The actual MoE large model inference optimization device can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements.

[0135] In addition, the technical effects of the MoE large model inference optimization device 310 can refer to the technical effects of the MoE large model inference optimization method based on dynamic scheduling of expert modules described in the above method embodiments, which will not be described here.

[0136] It should be understood that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), ready-to-program gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor, etc.

[0137] It should also be understood that the memory in the embodiments of the present application can be volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. Where the nonvolatile memory is, for example, read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. The volatile memory, which can be used as external cache, can be, for example, random access memory (RAM). By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0138] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0139] It should be understood that the term "and / or" herein merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after it are in an "or" relationship, but it can also represent an "and / or" relationship, which can be understood according to the context before and after it.

[0140] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0141] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined according to their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0142] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the devices, apparatuses and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0144] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0146] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0147] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0148] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A MoE large model inference optimization method based on expert module dynamic scheduling, characterized in that, The method comprises: uploading the MoE large model to a computer, reading a current hardware resource state, obtaining a GPU expert index list, a CPU expert index list, and a GPU candidate expert quantity value set; obtain a dialogue text dataset; based on the GPU expert index list and the GPU candidate expert quantity value set, pre-inference is performed on the dialogue text dataset by the MoE large model to obtain a first evaluation score set and a first expert index list; based on the GPU expert index list and the first expert index list, the GPU candidate expert quantity value set is optimized in value by an expert cache hit rate calculation method to obtain a second expert index list, a third expert index list, and a GPU candidate expert optimized value set; based on a relationship curve of an expert score rising value-transferred expert quantity, the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list, and the GPU expert index list are used for CPU expert module dynamic scheduling to obtain a GPU unloaded expert quantity set and a CPU expert scheduling index list; based on the GPU unloaded expert quantity set, the first evaluation score set, the third expert index list, and the CPU expert scheduling index list are used for GPU expert module dynamic supplement to obtain a GPU expert supplement index list; based on the second expert index list, the CPU expert scheduling index list, and the GPU expert supplement index list, a scheduled GPU expert index list is obtained; based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, re-inference is performed on the dialogue text dataset by the MoE large model to obtain an inference result text dataset.

2. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, The MoE large model comprises a plurality of decoding neural network layers; The decoding neural network layer comprises an attention layer, a feedforward neural network layer, and a normalization layer; The feedforward neural network layer comprises a gating function and a plurality of expert modules; the plurality of expert modules are a plurality of neural network modules with the same structure and different parameters.

3. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 2, characterized in that, The pre-inference based on the GPU expert index list and the GPU candidate expert quantity value set on the dialogue text dataset by the MoE large model to obtain the first evaluation score set and the first expert index list comprises: based on the GPU candidate expert quantity value set and the GPU expert index list, a dialogue hidden state vector set is obtained by performing token-by-token hidden state vector extraction on the dialogue text dataset; based on the dialogue hidden state vector set, expert module evaluation is performed by the gating function of the plurality of feedforward neural network layers to obtain the first evaluation score set; based on descending arrangement of the first evaluation score set, the first expert index list is generated by selecting the first k expert modules.

4. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, The value optimization of the preset GPU candidate expert quantity value set by the expert cache hit rate calculation method to obtain the second expert index list, the third expert index list, and the GPU candidate expert optimized value set comprises: According to the GPU expert index list, a second expert index list loaded on the GPU and used in the inference process and a third expert index list loaded on the GPU and not used in the inference process are obtained by comparison and screening according to the first expert index list; According to the first expert index list and the second expert index list, a set of GPU expert cache hit rates is obtained by calculation; According to the GPU expert index list, the number of loaded experts on the GPU is obtained; According to the set of expert cache hit rates and the number of loaded experts on the GPU, a GPU candidate expert optimization value set is obtained by calculation.

5. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 4, characterized in that, The calculation method of the GPU candidate expert optimization value set is as follows formula (1): （1）； Wherein, L is the total number of decoding neural network layers in the MoE large model; C is the number of loaded experts of the GPU; is the optimized value of the expert module in the ith decoding neural network layer; is the expert cache hit rate of the ith decoding neural network layer.

6. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, According to the first expert index list, the second expert index list, the first evaluation score set, the CPU expert index list and the GPU expert index list, a GPU unloaded expert number set and a CPU expert scheduling index list are obtained by CPU expert module dynamic scheduling based on the expert score increase value-transferred expert number relationship curve, including: According to the first expert index list and the second expert index list, a set of GPU unloaded expert numbers is obtained by difference set calculation; and a candidate expert value threshold set is determined according to the set of GPU unloaded expert numbers; Based on the candidate expert value threshold set, a second evaluation score set under different candidate expert numbers is obtained by using the MoE large model to perform transmission granularity scanning according to the dialogue text data set; According to the second evaluation score set, an expert score increase value set is obtained by calculation; According to the expert score increase value set, data analysis is performed and a set of expert score increase value-transferred expert number relationship curves is drawn; Based on the preset transferred expert number threshold, according to the set of relationship curves, the candidate expert number set corresponding to the maximum slope is selected as the transferred expert number set; In the CPU expert index list, according to the first evaluation score set, a number of expert modules corresponding to the transferred expert number set are selected to obtain the CPU expert scheduling index list.

7. The MoE large model inference optimization method based on dynamic scheduling of expert modules according to claim 1, characterized in that, According to the first evaluation score set, the third expert index list and the CPU expert scheduling index list, a GPU expert supplement index list is obtained by GPU expert module dynamic supplement based on the set of GPU unloaded expert numbers, including: According to the CPU expert scheduling index list, a CPU expert scheduling number set is obtained; According to the set of GPU unloaded expert numbers and the CPU expert scheduling number set, a GPU expert supplement number set is obtained by calculation; In the third expert index list, according to the first evaluation score set, a number of expert modules corresponding to the GPU expert supplement number set are selected to obtain the GPU expert supplement index list.

8. An apparatus for MoE large model inference optimization based on expert module dynamic scheduling, the apparatus being configured to implement the method for MoE large model inference optimization based on expert module dynamic scheduling according to any one of claims 1-7. The device comprises: A hardware information reading module is configured to upload the MoE large model to a computer, read the current hardware resource state, and obtain a GPU expert index list, a CPU expert index list and a GPU candidate expert number value set; The first dialogue reasoning module is configured to obtain dialogue text data sets; based on a GPU expert index list and a GPU candidate expert quantity value set, first evaluation score sets and a first expert index list are obtained by pre-reasoning through a MoE large model according to the dialogue text data sets; The expert value optimization module is configured to perform value optimization on the GPU candidate expert quantity value set through an expert cache hit rate calculation method according to the GPU expert index list and the first expert index list, and obtain a second expert index list, a third expert index list and a GPU candidate expert optimized value set; The CPU expert scheduling module is configured to perform CPU expert module dynamic scheduling according to the first expert index list, the second expert index list, the first evaluation score set, a CPU expert index list and a GPU expert index list based on a relationship curve of expert score rising value-transmission expert quantity, and obtain a GPU unloaded expert quantity set and a CPU expert scheduling index list; The GPU expert supplement module is configured to perform GPU expert module dynamic supplement according to the first evaluation score set, the third expert index list and the CPU expert scheduling index list based on the GPU unloaded expert quantity set, and obtain a GPU expert supplement index list; The second dialogue reasoning module is configured to obtain a scheduled GPU expert index list according to the second expert index list, the CPU expert scheduling index list and the GPU expert supplement index list; and perform re-reasoning through a MoE large model according to dialogue text data sets based on the scheduled GPU expert index list and the GPU candidate expert optimized value set, and obtain a reasoning result text data set.

9. A MoE large model inference optimization device, characterized in that, The MoE large model reasoning optimization equipment comprises: a processor; a memory, wherein the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the method in any one of claims 1 to 7.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores program codes, and the program codes can be called and executed by the processor to implement the method in any one of claims 1 to 7.

Images