Hybrid expert model reasoning method and device

Abstract

The invention discloses a method and a device for reasoning a hybrid expert model, which are used for solving the technical problem of low reasoning efficiency caused by the existing reasoning process based on the hybrid expert model. The method comprises the steps of obtaining a pre-running text data set, performing expert quantization on a preset hybrid expert model, and determining a quantized hybrid expert model; adopting a preset torch function to adjust an initial memory environment corresponding to the quantitative hybrid expert model according to the model configuration file of the quantitative hybrid expert model, and determining a target memory environment; in the target memory environment, inputting the pre-running text data set into the quantitative hybrid expert model for pre-running reasoning, and determining the average time length of model calculation; adopting a preset expert prefetching inequality to calculate the average duration according to the model, and calculating the number of prefetchable experts; and in the target memory environment, based on the number of prefetchable experts, reasoning the to-be-reasoned text data by adopting the quantitative mixed expert model, and outputting a model reasoning result.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a hybrid expert model reasoning method and device. Background Art

[0002] In recent years, with the rapid development of deep learning, large models have achieved remarkable results in various fields. For better data privacy, availability and personalization of AI functions, large models have gradually been applied to edge devices such as personal PCs, smartphones and even the Internet of Things. For example, Qualcomm has deployed a large model of Wenshengtu with more than 1 billion parameters on smartphones. Huawei has also embedded multimodal models in its smartphones to facilitate accurate content search based on natural language.

[0003] In order to meet the performance and power consumption requirements of the end-side equipment, the industry believes that 3B is the golden size of the end-side. However, compared with the dense model running 3B, the sparse MoE model (Mixture-of-Experts) can provide better reasoning services when the same number of parameters is activated. For example, the Qwen1.5-MoE model only activates 2.7B parameters, and the reasoning effect and speed are better than various 7B models.

[0004] However, although the number of activated parameters of the MoE model is low, the overall number of model parameters is high. For example, although Qwen1.5-MoE only activates 2.7B parameters, it has 14.3B parameters and requires about 30GB of memory space to run reasoning. In contrast, end-side devices often have only a small memory space. For example, NVIDIA's Jetson Orin Nano development board, as a common device for embedded development, has only 8GB or 16GB of memory. At present, most models of major mobile phone manufacturers have less than or equal to 16GB of memory.

[0005] Most of the existing reasoning processes based on hybrid expert models train a hash network (small model) to predict the experts that will be selected at each layer, so as to pre-fetch these experts in advance to complete the reasoning. However, every time a new model is launched, additional training costs are required to train the corresponding small model for prediction, resulting in excessively high application costs. Summary of the invention

[0006] The present invention provides a hybrid expert model reasoning method and device, which are used to solve the technical problem of low reasoning efficiency caused by the existing reasoning process based on the hybrid expert model.

[0007] A first aspect of the present invention provides a hybrid expert model reasoning method, comprising:

[0008] Obtain a pre-run text data set, perform expert quantization on a preset hybrid expert model, and determine a quantized hybrid expert model;

[0009] Using a preset torch function to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model, and determine the target memory environment;

[0010] In the target memory environment, inputting the pre-run text data set into the quantized hybrid expert model for pre-run reasoning to determine the average model calculation duration;

[0011] The average duration is calculated according to the model using the pre-set expert pre-fetch inequality, and the number of pre-fetchable experts is calculated;

[0012] In the target memory environment, based on the number of pre-fetchable experts, the quantized hybrid expert model is used to reason the text data to be reasoned, and a model reasoning result is output.

[0013] Optionally, the using a preset torch function to adjust the initial memory environment corresponding to the quantized hybrid expert model according to a model configuration file of the quantized hybrid expert model to determine the target memory environment includes:

[0014] Performing memory recognition on the initial memory environment corresponding to the quantized hybrid expert model based on a preset torch function, and outputting a memory budget;

[0015] Based on the model configuration file, determining the shape and size of the network layer corresponding to the quantized hybrid expert model;

[0016] According to the shape and size of the network layer, calculate the memory size occupied by the network layer;

[0017] Performing a subtraction operation on the memory budget and the memory size occupied by the network layer, and outputting a remaining space budget;

[0018] Performing a division operation on the remaining space budget and the memory size occupied by a single expert in the memory size occupied by the network layer to determine the total number of cacheable experts;

[0019] Calculating the number of deep expert caches according to the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model;

[0020] Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, an initial memory environment corresponding to the quantized hybrid expert model is adjusted to determine a target memory environment.

[0021] Optionally, the quantized hybrid expert model includes an input embedding layer, a plurality of target hybrid expert modules and an output embedding layer; based on the number of pre-fetchable experts, the quantized hybrid expert model is used to perform reasoning on the text data to be reasoned, and the model reasoning result is output, including:

[0022] Perform feature mapping on the text data to be inferred through an input embedding layer, and output a text vector representation;

[0023] Based on the number of pre-fetchable experts, multiple target mixed expert modules are used to perform mixed expert calculation on the text vector representation to generate a target intermediate state;

[0024] The target intermediate state is input into the output embedding layer for feature mapping, and the model inference result is output.

[0025] Optionally, the target mixture expert module includes a gating layer, an expert layer, an attention layer and a normalization layer; the data processing process of the adjacent target mixture expert module includes:

[0026] Normalizing the input vector data of the first target mixture expert module through the normalization layer of the first target mixture expert module, outputting a first initial normalized vector, and performing a linear transformation on the first initial normalized vector through the attention layer of the first target mixture expert module to generate a first linear vector;

[0027] Normalizing the first linear vector using a normalization layer of the first target hybrid expert module to generate a first intermediate normalized vector;

[0028] Performing weight calculation according to the first intermediate normalized vector through the gating layer of the first target hybrid expert module to determine first weight values ​​corresponding to multiple experts in the expert layer of the first target hybrid expert module;

[0029] Sorting the first weight values ​​in descending order, selecting experts corresponding to the first weight values ​​of the number of pre-fetchable experts as first activated experts, and dequantizing the first activated experts to determine a plurality of first dequantized activated experts;

[0030] Using each of the first dequantization activation experts to perform expert calculations on the first intermediate normalized vector, outputting expert results corresponding to each of the first dequantization activation experts, and merging the expert results corresponding to each of the first dequantization activation experts to generate a first intermediate state;

[0031] Normalizing the first intermediate state through a normalization layer of a second target mixture expert module to output a second initial normalized vector, and performing a linear transformation on the second initial normalized vector through an attention layer of the second target mixture expert module to generate a second linear vector;

[0032] Normalizing the second linear vector using a normalization layer of the second target mixture expert module to generate a second intermediate normalized vector;

[0033] Using the gating layer of the second target mixed expert module to perform weight calculation on the first intermediate normalized vector, and determining second weight values ​​corresponding to multiple experts in the expert layer of the second target mixed expert module;

[0034] Sorting the second weight values ​​in descending order, selecting experts corresponding to the second weight values ​​of the number of pre-fetchable experts as second activated experts, and dequantizing the second activated experts to determine a plurality of second dequantized activated experts;

[0035] The second dequantization activation experts are used to perform expert calculations on the second intermediate normalized vectors respectively, and the expert results corresponding to the second dequantization activation experts are output. The expert results corresponding to the second dequantization activation experts are merged to generate a second intermediate state.

[0036] Optionally, the model calculation average duration includes a single expert transmission duration, an attention layer calculation average duration, a gating layer calculation average duration, an expert layer calculation average duration, and a normalization layer calculation average duration; the preset expert pre-fetch inequality is specifically:

[0037]

[0038] Where x is the initial number of pre-fetchable experts; transmission time 单个专家 The transfer time of a single expert, which means the time required to load a single expert from the hard disk to the memory; calculation time 注意力层 Calculate the average duration for the attention layer; calculate the duration 专家层 Calculate the average duration for the expert layer; calculate the duration 归一化层 Calculate the average duration for the normalization layer; Calculate the duration 门控层 Calculate the average duration for the gated layer.

[0039] Optionally, the network layer data also includes the number of experts included in the target hybrid expert module and the number of layers of the quantized hybrid expert model; the calculation process of the number of deep expert caches is specifically as follows:

[0040] ;

[0041] The number of caches深层 Cache quantity for deep experts; Cache quantity 总 is the total number of cacheable experts; the number of fully cached layers is the number of fully cached layers of shallow experts; the number of experts contained in each layer is the number of experts contained in the target hybrid expert module; the total number of layers is the number of layers of the quantized hybrid expert model.

[0042] A second aspect of the present invention provides a hybrid expert model reasoning device, comprising:

[0043] An acquisition module is used to acquire a pre-running text data set, and to perform expert quantization on a preset hybrid expert model to determine a quantized hybrid expert model;

[0044] An adjustment module, configured to adjust an initial memory environment corresponding to the quantized hybrid expert model according to a model configuration file of the quantized hybrid expert model by using a preset torch function, and determine a target memory environment;

[0045] A pre-running reasoning module is used to input the pre-running text data set into the quantized hybrid expert model in the target memory environment to perform pre-running reasoning and determine the average model calculation time;

[0046] A calculation module, used for calculating the average duration according to the model by using a preset expert pre-fetching inequality, and calculating the number of pre-fetchable experts;

[0047] The output module is used to, in the target memory environment, based on the number of pre-fetchable experts, use the quantized hybrid expert model to reason on the text data to be reasoned, and output the model reasoning result.

[0048] A third aspect of the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the hybrid expert model reasoning method as described in any one of the above items.

[0049] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the steps of the hybrid expert model reasoning method as described in any one of the above items are implemented.

[0050] A fifth aspect of the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer executes the steps of the hybrid expert model reasoning method as described in any one of the above items.

[0051] It can be seen from the above technical solutions that the present invention has the following advantages:

[0052] The above technical scheme of the present invention provides a hybrid expert model reasoning method, firstly, a pre-run text data set is obtained, and an expert quantization is performed on a preset hybrid expert model to determine a quantized hybrid expert model; then, a preset torch function is used to adjust the initial memory environment corresponding to the quantized hybrid expert model according to a model configuration file of the quantized hybrid expert model to determine a target memory environment; in the target memory environment, the pre-run text data set is input into the quantized hybrid expert model for pre-run reasoning to determine the average model calculation time; a preset expert pre-fetch inequality is used to calculate the number of pre-fetchable experts according to the average model calculation time; finally, in the target memory environment, based on the number of pre-fetchable experts, a quantized hybrid expert model is used to reason on the text data to be reasoned, and a model reasoning result is output; based on the above scheme, a preset torch function is used to complete the adjustment of the memory environment according to the model configuration file of the quantized hybrid expert model to obtain the target memory environment, and the number of pre-fetchable experts is calculated according to the pre-run reasoning process, and in combination with the target memory environment and the number of pre-fetchable experts, a quantized hybrid expert model is used to reason on the text data to be reasoned, and a process of outputting the model reasoning result is performed. The present invention does not need to introduce additional training costs to train corresponding small models for prediction, thereby reducing application costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A flowchart of a hybrid expert model reasoning method provided in Example 1 of the present invention;

[0055] Figure 2 A reasoning framework diagram of the hybrid expert model reasoning method provided in the first embodiment of the present invention;

[0056] Figure 3 A schematic diagram of the routing weights during prediction provided by the first embodiment of the present invention;

[0057] Figure 4 A schematic diagram of routing weights during actual calculation provided by the first embodiment of the present invention;

[0058] Figure 5 A schematic diagram of the average pre-fetch accuracy of each layer provided in the first embodiment of the present invention;

[0059] Figure 6 A schematic diagram of the process of cross-layer expert prefetching provided in the first embodiment of the present invention;

[0060] Figure 7 A schematic diagram comparing the traditional Transformer model and the MoE model provided in the first embodiment of the present invention;

[0061] Figure 8 This is a structural block diagram of a hybrid expert model reasoning device provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0062] The embodiment of the present invention provides a hybrid expert model reasoning method and device, which are used to solve the technical problem of low reasoning efficiency caused by the existing reasoning process based on the hybrid expert model.

[0063] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Terminology explanation:

[0065] Mixture-of-Experts (MoE): The Mixture-of-Experts model replaces the feedforward neural network layer in the Transformer with the MoE layer. The MoE layer consists of a gating network and multiple experts. The gating function is the key feature of the MoE, which discretely decides the expert selection for each token. The most common practice is to select the top k experts for each token. In this case, the gating function is usually a softmax function, which activates the top-k experts to participate in the calculation according to the calculated probability distribution, and k is generally 1 or 2. Experts are essentially feedforward neural networks (FFN).

[0066] Token: In natural language processing, a token usually refers to a meaningful piece of text. When processing text, large models will split the input sentence into tokens.

[0067] Pipeline: The pipeline in a computer breaks down a task into multiple independent stages, each of which can process different tasks in parallel. The IO-computing pipeline below mainly describes the processing order and association of calculation and data transmission in parallel.

[0068] B: billion, a unit of measurement for large model parameters. For example, 3B below means the model has 3 billion parameters.

[0069] Model reasoning is divided into prefill and decoding stages:

[0070] Prefill: In this phase, the model receives a complete input sequence and performs calculations at once. Through this calculation, the model generates and caches all necessary intermediate states (such as KV cache) to provide a basis for subsequent step-by-step generation. It is usually used for initial context understanding of the sequence, especially for reasoning about long sequences.

[0071] Decoding: In the decoding phase, the model uses the KV cache cached in the pre-filling phase to generate step-by-step. At each step, the model predicts the next token based on the output of the previous step and the cached state, and updates the KV cache and other states until a complete output sequence is generated. This process is carried out step by step and is suitable for tasks such as dialogue generation and text continuation.

[0072] MMLU (Massive Multitask Language Understanding): A large-scale benchmark dataset designed for evaluating the multi-task learning capabilities of language models. It covers 57 tasks covering a wide range of subject areas, including humanities, science, mathematics, etc., and the difficulty of tasks ranges from high school level to expert level. This dataset is designed to test the generalization ability of the model in various tasks, especially those involving reasoning, knowledge application, and multi-domain problem solving.

[0073] CUDA Stream: It is a concept in CUDA programming, which refers to a series of operation queues executed sequentially on the GPU. Each CUDA Stream can execute tasks independently, allowing different computing and data transfer operations to overlap, thereby achieving parallelization and higher resource utilization. By assigning tasks to multiple Streams, the program can improve performance and reduce waiting time, especially in the case of mixed computing and memory transfer, there is a significant optimization effect.

[0074] See also Figure 1 , Figure 1 A flowchart of the steps of a hybrid expert model reasoning method provided in Example 1 of the present invention.

[0075] The present invention provides a hybrid expert model reasoning method, comprising:

[0076] Step 101: obtain a pre-running text data set, and perform expert quantization on a preset hybrid expert model to determine a quantized hybrid expert model.

[0077] Please note that Figure 2,The reasoning framework of the present invention can be divided into two stages: deployment stage and reasoning stage. ,The deployment stage includes model quantization and expert caching (memory environment adjustment), and the reasoning stage includes expert ,management and pipeline management.

[0078] Furthermore, for model quantization, quantization is an indispensable part of edge-side reasoning because it can greatly reduce the demand for memory and shorten the IO time. Since sparsely activated experts account for the vast majority of the parameters of the MoE model, and other parts except experts will participate in the calculation during each round of reasoning, the present invention chooses to store all parts except experts, namely the attention layer, normalization layer, embedding layer, gating layer, etc. in the memory according to the original precision, and then only unloads and quantizes the experts.

[0079] Furthermore, after weighing the speed and accuracy of various quantization algorithms, the present invention chooses to use semi-quadratic quantization to quantize the expert part. In order to select the appropriate number of quantization bits, the present invention uses semi-quadratic quantization to quantize the Qwen-1.5-MoE model, and uses the MMLU data set for reasoning, and the calculated accuracy loss is shown in Table 1. Obviously, INT4 quantization greatly reduces memory requirements while almost maintaining the original reasoning quality of the model.

[0080] Table 1 The impact of semi-quadratic quantization on model accuracy

[0081]

[0082] In this embodiment, the present invention uses semi-quadratic quantization to perform INT4 quantization on the expert part of the MoE model, and does not quantize the part other than the expert, and always keeps it in the memory. In addition, the present invention uses a dequantization algorithm to convert each expert into its original data type before it is used.

[0083] Step 102: Use a preset torch function to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model to determine the target memory environment.

[0084] It should be noted that if as many experts as possible can be cached in the memory under the existing memory limit, it not only reduces the IO between the memory and the hard disk, but also increases the possibility of expert hits (hitting means that the expert selected by the token is in the memory without waiting for IO). Moreover, the quantified experts have lower memory requirements, which helps to cache more experts in a limited memory space. The existing expert cache strategy often chooses the least recently used cache strategy (LRU, Least Recently Used) or the least frequently used cache strategy (LFU, Least Frequently Used). The two count the least recently used experts and the least frequently used experts respectively, and evict them from the memory. Therefore, the present invention uses an adaptive cache replacement algorithm (ARC, Adaptive Replacement Cache), which can dynamically balance the two cache strategies of LRU and LFU. Specifically, the present invention maintains two lists: one for tracking the most recently used cache items, and the other for tracking the frequently used cache items. ARC can automatically adjust the size of the two lists according to the changes in the access mode, so as to adapt to different workloads and provide a higher cache hit rate.

[0085] Further, step 102 may include the following sub-steps S21-S27:

[0086] S21, performing memory recognition on the initial memory environment corresponding to the quantized hybrid expert model based on a preset torch function, and outputting a memory budget;

[0087] S22. Based on the model configuration file, determine the shape and size of the network layer corresponding to the quantitative hybrid expert model;

[0088] S23. Calculate the memory size occupied by the network layer according to the shape and size of the network layer;

[0089] S24, performing a subtraction operation on the memory budget and the memory size occupied by the network layer, and outputting the remaining space budget;

[0090] S25, dividing the remaining space budget and the memory size occupied by a single expert in the network layer memory size to determine the total number of cacheable experts;

[0091] Network layer shape and size is the shape and size of each layer in the quantized mixture of experts model.

[0092] The memory size occupied by the network layer includes the memory size occupied by a single expert, the memory size occupied by the attention layer, the memory size occupied by the normalization layer, the memory size occupied by the embedding layer, and the memory size occupied by the gating layer.

[0093] It should be noted that in order to dynamically cache a large number of experts in different environments, the present invention first uses the function in the torch library (preset torch function) to identify the memory budget of the current environment. Then, the model configuration file is used to obtain the shape and size of each layer (network layer shape and size) to calculate the memory size occupied by a single expert, attention layer, normalization layer, embedding layer, and gating layer, that is, the memory size occupied by the network layer. Next, the total memory size of the sum of the memory sizes occupied by the attention layer, normalization layer, embedding layer, and gating layer is subtracted from the memory budget to obtain the remaining space budget. Then, the remaining space budget is divided by the memory size occupied by a single expert to obtain the total number of experts that can be cached in the current environment, that is, the total number of experts that can be cached.

[0094] S26, calculating the number of deep expert caches according to the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model;

[0095] S27. Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, the initial memory environment corresponding to the quantized hybrid expert model is adjusted to determine the target memory environment.

[0096] The network layer data includes the number of shallow expert full cache layers, the number of experts included in the target hybrid expert module, and the number of layers of the quantized hybrid expert model.

[0097] It should be noted that the existing practice is to use the total number of cacheable experts divided by the number of model layers to obtain the average number of cacheable experts per layer, but this will result in a lower accuracy rate for shallow expert pre-fetching and a higher accuracy rate for deep expert pre-fetching. Therefore, in order to compensate for the low accuracy of the shallow layer, the present invention chooses to cache all shallow experts, and then the remaining cacheable number is evenly divided among subsequent layers. Among them, the number of fully cached layers varies with the memory budget; assuming that the number of layers of the quantized hybrid expert model is 24, and all experts in the first three layers are regarded as shallow experts, the number of fully cached layers of shallow experts is 3.

[0098] Furthermore, the calculation process of the number of deep expert caches is as follows:

[0099] ;

[0100] The number of caches 深层 Cache quantity for deep experts; Cache quantity 总 is the total number of cacheable experts; the number of fully cached layers is the number of fully cached layers of shallow experts; the number of experts contained in each layer is the number of experts contained in the target hybrid expert module; the total number of layers is the number of layers of the quantized hybrid expert model.

[0101] It is worth mentioning that if the total number of experts that can be cached is not enough to support multiple shallow caches, the number of deep caches will be 0, and as many shallow caches as possible will be placed starting from the first layer.

[0102] Step 103: In the target memory environment, the pre-run text data set is input into the quantitative hybrid expert model for pre-run reasoning to determine the average model calculation time.

[0103] The average calculation time of the model includes the single expert transmission time, the average calculation time of the attention layer, the average calculation time of the gating layer, the average calculation time of the expert layer and the average calculation time of the normalization layer.

[0104] It should be noted that the specific number of experts to pre-fetch x is difficult to determine because it is related to the calculation time t between the two gating layers. If the total transmission time of x experts is much less than t, the opportunity to improve accuracy is wasted. If the total transmission time of x experts is greater than t, it will affect the calculation efficiency and increase the delay of reasoning. Therefore, in order to realize a pipeline that ensures complete overlap between calculation and communication, the present invention uses modeling-based pipeline management.

[0105] Furthermore, after completing the model deployment, the present invention records the average calculation time of the attention layer, gating layer, expert layer, normalization layer, etc. based on the pre-run reasoning process, and also records the time required to load a single expert from the hard disk to the memory, that is, the average model calculation time.

[0106] Step 104: Use the preset expert pre-fetching inequality to calculate the average duration according to the model and calculate the number of pre-fetchable experts.

[0107] It should be noted that the preset expert pre-fetches the inequality, specifically:

[0108]

[0109] Where x is the initial number of pre-fetchable experts; transmission time 单个专家 The transfer time of a single expert, which means the time required to load a single expert from the hard disk to the memory; calculation time 注意力层 Calculate the average duration for the attention layer; calculate the duration 专家层 Calculate the average duration for the expert layer; calculate the duration 归一化层 Calculate the average duration for the normalization layer; Calculate the duration 门控层 Calculate the average duration for the gated layer.

[0110] Furthermore, the initial number of pre-fetchable experts x calculated by the inequality is rounded down to obtain the final number of transmittable experts |x|, that is, the number of pre-fetched experts. Since the present invention uses semi-quadratic quantization to quantize each expert into INT4 during the deployment phase, the transmission time is 单个专家Substantially reduced, so that x is further improved. Secondly, the present invention uses a cache algorithm, so not all |x| will have transmission requirements, further facilitating the complete overlap of computing and communication.

[0111] Step 105: In the target memory environment, based on the number of pre-fetchable experts, a quantized hybrid expert model is used to infer the text data to be inferred, and the model inference result is output.

[0112] The quantized mixture of experts model includes an input embedding layer, multiple target mixture of experts modules and an output embedding layer.

[0113] Specifically, step 105 may include the following sub-steps S51-S53:

[0114] Step S51, feature mapping is performed on the text data to be inferred through the input embedding layer, and a text vector representation is output;

[0115] Step S52: Based on the number of pre-fetchable experts, multiple target hybrid expert modules are used to perform hybrid expert calculation on the text vector representation to generate a target intermediate state;

[0116] Step S53: input the target intermediate state into the output embedding layer for feature mapping, and output the model inference result.

[0117] It should be noted that after the model deployment is completed, as the request is input, the present invention dynamically manages the experts to achieve a completely overlapping pipeline and eliminate the impact of expert offloading on reasoning. When reasoning MoE on the end side, the most critical issue is how to pre-fetch experts. Because only accurate pre-fetching of experts can achieve the overlap of computing and communication to reduce reasoning latency.

[0118] Furthermore, the inventor tested the cosine similarity between the inputs of the two gating layers and found that the cosine similarity between the two was as high as 88.83%. Then, the inventor continued to calculate the accuracy of this pre-fetching method and found that the accuracy can reach 99.75% in the pre-filling stage and 78.79% in the decoding stage. It is very ideal that this pre-fetching method can achieve such a high accuracy without introducing any additional cost.

[0119] For further information, see Figure 3-Figure 4 In order to further improve the pre-fetching accuracy, the present invention explores the routing weights calculated by the gating layer. In the calculation process of the gating layer, a corresponding weight is assigned to each expert according to the input, and then the k experts with the highest weights are selected for activation. Figure 3-Figure 4 As shown, it is the experimental result based on a gating layer in the Qwen-1.5-MoE model. The gating layer of the Qwen-1.5-MoE model will select 4 out of 60 experts. Figure 3The expert weights in are obtained through cross-layer calculation. According to this result, experts 5, 11, 21, and 36 should be pre-fetched. However, the actual calculation result of the gating layer is as follows Figure 4 As shown in Figure 2, it is clear that the experts who will actually be activated are 5, 21, 31, and 36. However, comparing the two figures ( Figure 3 and Figure 4 ) It can be seen that the distribution of routing weights is extremely similar, and the expert 31 who failed to pre-fetch Figure 3 It ranks fifth in the prediction results. This means that if the number of expert prefetches is increased, the accuracy of expert prefetches can be greatly improved.

[0120] For further information, see Figure 5 , the inventors further conducted experiments and found that the pre-fetching accuracy of this method can reach 97.15%. Therefore, based on the above, the inventors send the input to the next gating layer while calculating at each gating layer, use the CPU to perform cross-layer calculations in parallel, and then pre-fetch the first |x| experts according to the calculation results (i.e., routing weights) to achieve cost-free and highly accurate expert pre-fetching. Among them, the inventors found that when calculating the accuracy of pre-fetching at each layer Figure 5 The phenomenon shown is that the shallow expert prefetching accuracy is lower than the deep one.

[0121] To do this, see Figure 6 The present invention proposes cross-layer expert prefetching, which sends the input of the gated layer in the current MoE block (block i) to the gated layer in the next MoE block (block i+1) for calculation. Since the calculation process can be completed by the CPU and prefetching starts immediately based on the calculation results, the IO process of the prefetch expert can overlap with the attention layer and expert layer calculation between the two gates, so it will not cause delay to the reasoning process.

[0122] Optionally, the target mixture expert module includes a gating layer, an expert layer, an attention layer and a normalization layer; the data processing process of the adjacent target mixture expert modules can be implemented by executing the following steps S01 to S010:

[0123] Step S01, normalizing the input vector data of the first target mixture expert module through the normalization layer of the first target mixture expert module, outputting a first initial normalized vector, and performing a linear transformation on the first initial normalized vector through the attention layer of the first target mixture expert module to generate a first linear vector;

[0124] Step S02: normalizing the first linear vector using the normalization layer of the first target hybrid expert module to generate a first intermediate normalized vector;

[0125] Step S03, performing weight calculation according to the first intermediate normalized vector through the gating layer of the first target mixed expert module, and determining first weight values ​​corresponding to multiple experts in the expert layer of the first target mixed expert module;

[0126] Step S04, sorting the first weight values ​​in descending order, selecting experts corresponding to the first weight values ​​of the number of experts that can be pre-fetched as first activated experts, and dequantizing the first activated experts to determine a plurality of first dequantized activated experts;

[0127] Step S05: using each first dequantization activation expert to perform expert calculation on the first intermediate normalized vector respectively, outputting the expert results corresponding to each first dequantization activation expert, and merging the expert results corresponding to each first dequantization activation expert to generate a first intermediate state;

[0128] Step S06, normalizing the first intermediate state through the normalization layer of the second target mixture expert module, outputting a second initial normalized vector, and performing a linear transformation on the second initial normalized vector through the attention layer of the second target mixture expert module to generate a second linear vector;

[0129] Step S07: normalizing the second linear vector using the normalization layer of the second target hybrid expert module to generate a second intermediate normalized vector;

[0130] Step S08: using the gating layer of the second target mixed expert module to perform weight calculation on the first intermediate normalized vector, and determining second weight values ​​corresponding to multiple experts in the expert layer of the second target mixed expert module;

[0131] Step S09, sorting the second weight values ​​in descending order, selecting experts corresponding to the second weight values ​​of the number of experts that can be pre-fetched as second activated experts, and dequantizing the second activated experts to determine a plurality of second dequantized activated experts;

[0132] Step S010: Use each second dequantization activation expert to perform expert calculation on the second intermediate normalized vector respectively, output the expert results corresponding to each second dequantization activation expert, and merge the expert results corresponding to each second dequantization activation expert to generate a second intermediate state.

[0133] The input vector data is the vector data input to the target hybrid expert module.

[0134] It should be noted that the gating layer in MoE module i (the first target mixed expert module) and the gating layer in MoE module i+1 (the second target mixed expert module) will simultaneously receive and process the first intermediate normalized vector, thereby obtaining the first weight values ​​corresponding to multiple experts in the expert layer of the first target mixed expert module and the second weight values ​​corresponding to multiple experts in the expert layer of the second target mixed expert module; according to each first weight value and each second weight value, the first activated expert and the second activated expert of the pre-fetched expert number are determined. For example, assuming that the calculated number of pre-fetched experts is 3, all the first weight values ​​and all the second weight values ​​are sorted in descending order, and the experts corresponding to the first 3 first weight values ​​are selected as the first activated experts, and the experts corresponding to the first 3 second weight values ​​are selected as the second activated experts.

[0135] Furthermore, after each expert completes the transmission, the present invention uses another CUDA Stream to parallel dequantize the expert to the original data type (such as BF16 / FP16, etc.) to reduce the precision loss. The dequantization calculation of each expert is performed by the CPU, overlapping with the model calculation and expert transmission, and also does not introduce additional delay, that is, the activated expert is dequantized to obtain the dequantized expert (the first dequantized activated expert and the second dequantized activated expert).

[0136] Furthermore, each first dequantization activation expert performs expert calculation on the first intermediate normalized vector, outputs the expert results corresponding to each first dequantization activation expert, and merges the expert results corresponding to each first dequantization activation expert to generate a first intermediate state. The first intermediate state is sequentially processed by the normalization layer-attention layer-normalization layer in the second target mixed expert module, and after the second intermediate normalized vector is output, each second dequantization activation expert performs expert calculation on the second intermediate normalized vector, outputs the expert results corresponding to each second dequantization activation expert, and merges the expert results corresponding to each second dequantization activation expert to generate a second intermediate state.

[0137] Furthermore, if the second target mixture expert module is not connected to the next target mixture expert module, the second intermediate state output by the second target mixture expert module is used as the target intermediate state; if the second target mixture expert module is connected to the next target mixture expert module, the next target mixture expert module connected to the second target mixture expert module is used as the new second target mixture expert module, the second target mixture expert module is used as the new first target mixture expert module, the second intermediate state is used as the new first intermediate state, the second intermediate normalized vector is used as the new first intermediate normalized vector, and step S06 is jumped to execute until the second target mixture expert module is not connected to the next target mixture expert module, and the second intermediate state output by the second target mixture expert module is used as the target intermediate state.

[0138] In this embodiment, the present invention achieves high-accuracy prefetching of experts in a simple and cost-free manner (cross-layer prefetching), and further improves the accuracy of prefetching by utilizing routing weights. In addition, the present invention also uses quantization and pipeline modeling to achieve complete overlap of communication and computing, almost eliminating the inference delay caused by model offloading. At the same time, the present invention provides higher-quality large-model inference services by deploying MoE models on the end side, and realizes a MoE inference framework for fast inference on the end side.

[0139] As a comparison of technical effects, we can refer to the existing technologies. For the memory optimization problem in the large model reasoning process, offloading is one of the current mainstream solutions. The tensors that are not currently involved in the calculation can be offloaded to the hard disk and loaded into the memory each time they are needed to greatly reduce the demand for memory. However, due to the physical condition that the current computing speed is generally faster than the transmission speed, offloading is often accompanied by serious IO overhead. In this regard, dense models can often pre-fetch the parameters of the next layer in advance to achieve the overlap of computing and IO. However, due to the dynamic activation of experts, the MoE model is difficult to pre-fetch accurately, and the IO overhead of loading the entire MoE layer is large, resulting in high latency. There are two main methods to solve this problem in the existing solutions: (1) Activation perception. According to the expert selection of the token in the previous layer or layers, the expert selection of the token in the next layer is predicted based on statistical data. (2) Prediction model. A small network is trained to predict the expert selection of the token in each layer. However, the prediction accuracy of method (1) is not satisfactory, and method (2) will introduce additional overhead and poor portability. A corresponding small model must be retrained for each new MoE model.

[0140] For further information, see Figure 7 ,The existing large models are mainly based on the Transformer structure, such as Figure 7(a) Since GShard introduced the novel hybrid expert structure, researchers have continuously explored its great potential in improving the performance of large language models. It has gradually become one of the mainstream structures of large language models. Popular models such as GPT-4, Gemini 1.5, and Mixtral 8x7B all use the MoE structure. Unlike large models based on the Transformer structure, the hybrid expert model uses the MoE layer to replace the feedforward neural network layer, such as Figure 7 (b) As shown. The MoE layer consists of a gating network and multiple experts. The gating function is the key feature of MoE, which discretely selects the experts involved in the calculation for each input. The most common practice is to select the top k experts for each token. In this case, the gating function is usually a softmax function, which activates the top-k experts according to the calculated probability distribution, and k is generally 1 or 2. Current methods have difficulty in accurately predicting the choice of experts.

[0141] Experts are essentially feed-forward neural networks that are sparsely activated based on the results of the gating function. It is this sparse activation property that makes it the most feasible approach to obtain larger models. For each input, the final output of the MoE layer is a weighted sum of the outputs from the selected experts.

[0142] To address the memory optimization problem during large model inference, offloading is one of the current mainstream solutions. The client side often only has two levels of storage space: memory and hard disk. In the process of model inference, since each tensor does not participate in the calculation throughout the process, unloading the tensors that are not currently involved in the calculation to the hard disk can free up a lot of memory space. So in theory, as long as the memory is large enough to hold the computing resources and the hard disk is large enough to offload other resources, model inference can be performed on the client side. However, in contrast, the offloading strategy is always accompanied by latency issues. Because the relevant tensors are required to exist in the memory during calculation, the speed of tensor transmission between the memory and the hard disk is often slower than the speed of SoC calculation due to the limitation of transmission bandwidth. The SoC can only wait for the tensor transmission to be completed before performing calculations, resulting in a waste of computing power.

[0143] Quantization technology reduces the storage and computing requirements of the model by converting high-precision parameters in the model (such as 32 / 16-bit floating point numbers, FP32 / FP16 / BF16, etc.) into low-precision representations (such as 8 bits or lower, INT8 / INT4 / INT2 / INT1, etc.). This can significantly reduce the model size and computing complexity while keeping the model performance basically unchanged. It is usually used for model reasoning acceleration and deployment on resource-constrained devices (such as end-side mobile devices).

[0144] In summary, the existing reasoning methods mainly identify the importance of different experts and quantize them with different digits, thereby caching more experts in memory and reducing the time consumption of parameter transmission. Experts are predicted by activation perception. Each time the attention layer is calculated, the predicted experts are asynchronously transmitted and dequantized. Since the experts have been quantized, the transmission time is greatly reduced, which makes it easier to overlap calculation and communication and reduce the reasoning delay. At the same time, a hash network (small model) is trained to predict the experts that will be selected by the input at each layer, so as to pre-fetch these experts in advance to achieve low-latency MoE model reasoning in a resource-constrained environment. In addition, by modifying the original structure of the model. The MoE model selects the experts activated in each layer through gating, but removes the gating function of this layer, adds a pre-gate to the previous layer, and retrains these gating networks. Therefore, during reasoning, the experts to be activated in the i+1 layer can be calculated in the i layer, and then the experts are preloaded asynchronously, overlapping the calculation of the i layer with the loading of the experts in the i+1 layer, reducing the reasoning delay.

[0145] It can be seen that the existing reasoning method is based on activating perceptual prediction experts. Experimental verification shows that the accuracy of pre-fetching is not satisfactory, which will seriously affect the reasoning delay. Moreover, the reasoning speed improvement in the decoding stage is mainly considered, and the more complex prefill stage is not considered; at the same time, it is based on a newly trained small network. Although this approach based on small model prediction experts has high accuracy, it introduces additional training costs. Moreover, this approach has very poor portability. Whenever a new model is launched, if you want to apply this technology, you have to train the corresponding small model for prediction, which is unrealistic. In addition, Solution 3 changes the original structure of the model.

[0146] In response to the above problems, the present invention proposes a hybrid expert model reasoning method to achieve high-throughput MoE reasoning in resource-constrained scenarios. Specifically, the present invention first analyzes the reasons for the inefficiency of the current MoE reasoning pipeline, and then proposes a multi-batch processing pipeline based on expert perception, which uses the characteristics of high computing and low IO of popular experts to arrange multiple batches of calculations to minimize inter-layer bubbles and intra-layer bubbles in the pipeline. In addition, the present invention also adaptively measures the hardware capabilities in the current environment to generate the optimal execution plan in the current environment. At the same time, the present invention focuses on the reasoning speed of the MoE end side. Compared with the prior art, the expert pre-fetching accuracy of the present invention is much higher than it, and the implementation is simpler, and the present invention is lower in cost, does not require additional training costs, and the pre-fetching method of the present invention can be directly used in the reasoning process of any MoE model, and the technical portability is far superior to the prior art. In summary, the present invention implements high-accuracy pre-fetching of experts in a simple and efficient way, greatly reduces the reasoning delay caused by expert offloading, and can realize the fast reasoning of the MoE model on the end side.

[0147] In an embodiment of the present invention, the present invention provides a hybrid expert model reasoning method, firstly, a pre-run text data set is obtained, and an expert quantization is performed on a preset hybrid expert model to determine a quantized hybrid expert model; then, a preset torch function is used to adjust the initial memory environment corresponding to the quantized hybrid expert model according to a model configuration file of the quantized hybrid expert model to determine a target memory environment; in the target memory environment, the pre-run text data set is input into the quantized hybrid expert model for pre-run reasoning to determine the average model calculation time; a preset expert pre-fetch inequality is used to calculate the number of pre-fetchable experts according to the average model calculation time; finally, in the target memory environment, based on the number of pre-fetchable experts, a quantized hybrid expert model is used to reason on the text data to be reasoned, and a model reasoning result is output; based on the above scheme, a preset torch function is used to complete the adjustment of the memory environment according to the model configuration file of the quantized hybrid expert model to obtain the target memory environment, and the number of pre-fetchable experts is calculated according to the pre-run reasoning process, and in combination with the target memory environment and the number of pre-fetchable experts, a quantized hybrid expert model is used to reason on the text data to be reasoned, and a process of outputting the model reasoning result is performed. In the present invention, no additional training cost is required to train the corresponding small model for prediction, thereby reducing the application cost.

[0148] See also Figure 8 , Figure 8 This is a structural block diagram of a hybrid expert model reasoning device provided in Embodiment 2 of the present invention.

[0149] The present invention provides a hybrid expert model reasoning device, comprising:

[0150] The acquisition module 801 is used to acquire a pre-running text data set, and perform expert quantization on a preset hybrid expert model to determine a quantized hybrid expert model;

[0151] An adjustment module 802 is used to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model by using a preset torch function to determine a target memory environment;

[0152] A pre-running reasoning module 803 is used to input the pre-running text data set into the quantitative hybrid expert model for pre-running reasoning in the target memory environment to determine the average model calculation time;

[0153] A calculation module 804 is used to calculate the average duration according to the model using a preset expert pre-fetching inequality and calculate the number of pre-fetchable experts;

[0154] The output module 805 is used to use the quantized hybrid expert model to perform reasoning on the text data to be reasoned based on the number of pre-fetchable experts in the target memory environment, and output the model reasoning result.

[0155] Furthermore, the adjustment module 802 is specifically configured to:

[0156] Based on the preset torch function, the initial memory environment corresponding to the quantized hybrid expert model is identified and the memory budget is output;

[0157] Based on the model configuration file, determine the shape and size of the network layer corresponding to the quantized hybrid expert model;

[0158] Calculate the memory size occupied by the network layer based on the shape and size of the network layer;

[0159] Subtract the memory budget from the memory size occupied by the network layer, and output the remaining space budget;

[0160] Divide the remaining space budget and the memory size occupied by a single expert in the network layer to determine the total number of cacheable experts;

[0161] Calculate the number of deep expert caches based on the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model;

[0162] Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, the initial memory environment corresponding to the quantized hybrid expert model is adjusted to determine the target memory environment.

[0163] Furthermore, the quantitative hybrid expert model includes an input embedding layer, a plurality of target hybrid expert modules and an output embedding layer; the output module 805 is specifically used for:

[0164] Perform feature mapping on the text data to be inferred through the input embedding layer and output the text vector representation;

[0165] Based on the number of pre-fetchable experts, multiple target hybrid expert modules are used to perform hybrid expert calculations on the text vector representation to generate target intermediate states;

[0166] The target intermediate state is input into the output embedding layer for feature mapping, and the model inference result is output.

[0167] Optionally, the target mixture expert module includes a gating layer, an expert layer, an attention layer and a normalization layer; and the data processing process of the adjacent target mixture expert module includes:

[0168] Normalizing the input vector data of the first target mixture expert module through the normalization layer of the first target mixture expert module, outputting a first initial normalized vector, and performing a linear transformation on the first initial normalized vector through the attention layer of the first target mixture expert module to generate a first linear vector;

[0169] Normalizing the first linear vector using a normalization layer of a first target mixture expert module to generate a first intermediate normalized vector;

[0170] Performing weight calculation according to the first intermediate normalized vector through the gating layer of the first target mixed expert module to determine first weight values ​​corresponding to multiple experts in the expert layer of the first target mixed expert module;

[0171] Sorting the first weight values ​​in descending order, selecting experts corresponding to the first weight values ​​of the number of experts that can be pre-fetched as first activated experts, and dequantizing the first activated experts to determine a plurality of first dequantized activated experts;

[0172] Using each first dequantization activation expert to perform expert calculation on the first intermediate normalized vector respectively, outputting the expert results corresponding to each first dequantization activation expert, and merging the expert results corresponding to each first dequantization activation expert to generate a first intermediate state;

[0173] Normalizing the first intermediate state through a normalization layer of a second target mixture expert module, outputting a second initial normalized vector, and performing a linear transformation on the second initial normalized vector through an attention layer of the second target mixture expert module to generate a second linear vector;

[0174] Normalizing the second linear vector using a normalization layer of a second target mixture expert module to generate a second intermediate normalized vector;

[0175] Using the gating layer of the second target mixed expert module to perform weight calculation on the first intermediate normalized vector, and determining second weight values ​​corresponding to multiple experts in the expert layer of the second target mixed expert module;

[0176] Sort the second weight values ​​in descending order, select experts corresponding to the second weight values ​​of the number of experts that can be pre-fetched as second activated experts, and dequantize the second activated experts to determine a plurality of second dequantized activated experts;

[0177] The second dequantization activation experts are used to perform expert calculations on the second intermediate normalized vector respectively, and the expert results corresponding to the second dequantization activation experts are output. The expert results corresponding to the second dequantization activation experts are merged to generate a second intermediate state.

[0178] Furthermore, the average calculation time of the model includes the transmission time of a single expert, the average calculation time of the attention layer, the average calculation time of the gating layer, the average calculation time of the expert layer and the average calculation time of the normalization layer; the expert pre-fetch inequality is preset, specifically:

[0179]

[0180] Where x is the initial number of pre-fetchable experts; transmission time 单个专家 The transfer time of a single expert, which means the time required to load a single expert from the hard disk to the memory; calculation time 注意力层 Calculate the average duration for the attention layer; calculate the duration 专家层 Calculate the average duration for the expert layer; calculate the duration 归一化层 Calculate the average duration for the normalization layer; Calculate the duration 门控层 Calculate the average duration for the gated layer.

[0181] Furthermore, the network layer data also includes the number of experts included in the target hybrid expert module, the number of layers of the quantized hybrid expert model; the calculation process of the number of deep expert caches is specifically as follows:

[0182] ;

[0183] The number of caches 深层 Cache quantity for deep experts; Cache quantity 总 is the total number of cacheable experts; the number of fully cached layers is the number of fully cached layers of shallow experts; the number of experts contained in each layer is the number of experts contained in the target hybrid expert module; the total number of layers is the number of layers of the quantized hybrid expert model.

[0184] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0185] An embodiment of the present invention further provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the hybrid expert model reasoning method of any of the above embodiments.

[0186] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the hybrid expert model reasoning method in any of the above embodiments are implemented.

[0187] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the hybrid expert model reasoning method in any of the above embodiments when the computer program / instruction is executed by a processor.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid expert model reasoning method, characterized in that: include: Obtain a pre-run text data set, perform expert quantization on a preset hybrid expert model, and determine a quantized hybrid expert model; Using a preset torch function to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model, and determine the target memory environment; In the target memory environment, inputting the pre-run text data set into the quantized hybrid expert model for pre-run reasoning to determine the average model calculation duration; The average duration is calculated according to the model using the pre-set expert pre-fetch inequality, and the number of pre-fetchable experts is calculated; In the target memory environment, based on the number of pre-fetchable experts, the quantized hybrid expert model is used to reason the text data to be reasoned, and a model reasoning result is output.

2. The hybrid expert model reasoning method according to claim 1, characterized in that: The method of using a preset torch function to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model to determine the target memory environment includes: Performing memory recognition on the initial memory environment corresponding to the quantized hybrid expert model based on a preset torch function, and outputting a memory budget; Based on the model configuration file, determining the shape and size of the network layer corresponding to the quantized hybrid expert model; According to the shape and size of the network layer, calculate the memory size occupied by the network layer; Performing a subtraction operation on the memory budget and the memory size occupied by the network layer, and outputting a remaining space budget; Performing a division operation on the remaining space budget and the memory size occupied by a single expert in the memory size occupied by the network layer to determine the total number of cacheable experts; Calculating the number of deep expert caches according to the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model; Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, an initial memory environment corresponding to the quantized hybrid expert model is adjusted to determine a target memory environment.

3. The hybrid expert model reasoning method according to claim 1, characterized in that: The quantized hybrid expert model includes an input embedding layer, a plurality of target hybrid expert modules and an output embedding layer; The method of using the quantized hybrid expert model to perform reasoning on the text data to be reasoned based on the number of pre-fetchable experts and outputting a model reasoning result includes: Perform feature mapping on the text data to be inferred through an input embedding layer, and output a text vector representation; Based on the number of pre-fetchable experts, multiple target mixed expert modules are used to perform mixed expert calculation on the text vector representation to generate a target intermediate state; The target intermediate state is input into the output embedding layer for feature mapping, and the model inference result is output.

4. The hybrid expert model reasoning method according to claim 3, characterized in that: The target mixed expert module includes a gating layer, an expert layer, an attention layer and a normalization layer; the data processing process of the adjacent target mixed expert module includes: Normalizing the input vector data of the first target mixture expert module through the normalization layer of the first target mixture expert module, outputting a first initial normalized vector, and performing a linear transformation on the first initial normalized vector through the attention layer of the first target mixture expert module to generate a first linear vector; Normalizing the first linear vector using a normalization layer of the first target hybrid expert module to generate a first intermediate normalized vector; Performing weight calculation according to the first intermediate normalized vector through the gating layer of the first target hybrid expert module to determine first weight values ​​corresponding to multiple experts in the expert layer of the first target hybrid expert module; Sorting the first weight values ​​in descending order, selecting experts corresponding to the first weight values ​​of the number of pre-fetchable experts as first activated experts, and dequantizing the first activated experts to determine a plurality of first dequantized activated experts; Using each of the first dequantization activation experts to perform expert calculations on the first intermediate normalized vector, outputting expert results corresponding to each of the first dequantization activation experts, and merging the expert results corresponding to each of the first dequantization activation experts to generate a first intermediate state; Normalizing the first intermediate state through a normalization layer of a second target mixture expert module to output a second initial normalized vector, and performing a linear transformation on the second initial normalized vector through an attention layer of the second target mixture expert module to generate a second linear vector; Normalizing the second linear vector using a normalization layer of the second target mixture expert module to generate a second intermediate normalized vector; Using the gating layer of the second target mixed expert module to perform weight calculation on the first intermediate normalized vector, and determining second weight values ​​corresponding to multiple experts in the expert layer of the second target mixed expert module; Sorting the second weight values ​​in descending order, selecting experts corresponding to the second weight values ​​of the number of pre-fetchable experts as second activated experts, and dequantizing the second activated experts to determine a plurality of second dequantized activated experts; The second dequantization activation experts are used to perform expert calculations on the second intermediate normalized vectors respectively, and the expert results corresponding to the second dequantization activation experts are output. The expert results corresponding to the second dequantization activation experts are merged to generate a second intermediate state.

5. The hybrid expert model reasoning method according to claim 1, characterized in that: The model calculation average duration includes a single expert transmission duration, an attention layer calculation average duration, a gating layer calculation average duration, an expert layer calculation average duration, and a normalization layer calculation average duration; the preset expert pre-fetch inequality is specifically: Where x is the initial number of pre-fetchable experts; transmission time 单个专家 The transfer time of a single expert, which means the time required to load a single expert from the hard disk to the memory; calculation time 注意力层 Calculate the average duration for the attention layer; calculate the duration 专家层 Calculate the average duration for the expert layer; calculate the duration 归一化层 Calculate the average duration for the normalization layer; Calculate the duration 门控层 Calculate the average duration for the gated layer.

6. The hybrid expert model reasoning method according to claim 2, characterized in that: The network layer data also includes the number of experts included in the target hybrid expert module and the number of layers of the quantized hybrid expert model; the calculation process of the number of deep expert caches is specifically as follows: ； The number of caches 深层 The number of caches for deep experts; the number of caches 总 is the total number of cacheable experts; the number of fully cached layers is the number of fully cached layers of shallow experts; the number of experts contained in each layer is the number of experts contained in the target hybrid expert module; the total number of layers is the number of layers of the quantized hybrid expert model.

7. A hybrid expert model reasoning device, characterized in that: include: An acquisition module is used to acquire a pre-running text data set, and to perform expert quantization on a preset hybrid expert model to determine a quantized hybrid expert model; An adjustment module, configured to adjust an initial memory environment corresponding to the quantized hybrid expert model according to a model configuration file of the quantized hybrid expert model by using a preset torch function, and determine a target memory environment; A pre-running reasoning module is used to input the pre-running text data set into the quantized hybrid expert model in the target memory environment to perform pre-running reasoning and determine the average model calculation time; A calculation module, used for calculating the average duration according to the model by using a preset expert pre-fetching inequality, and calculating the number of pre-fetchable experts; The output module is used to, in the target memory environment, based on the number of pre-fetchable experts, use the quantized hybrid expert model to reason on the text data to be reasoned, and output the model reasoning result.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the hybrid expert model reasoning method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the hybrid expert model reasoning method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, wherein when the program instructions are executed by a computer, the computer is caused to execute the hybrid expert model reasoning method as described in any one of claims 1-6.

Images