A hybrid expert model reasoning method and apparatus

Abstract

This invention discloses a hybrid expert model inference method and apparatus to address the technical problem of low inference efficiency caused by existing hybrid expert model-based inference processes. The method includes: acquiring a pre-running text dataset and performing expert quantization on a pre-set hybrid expert model to determine the quantized hybrid expert model; adjusting the initial memory environment corresponding to the quantized hybrid expert model using a pre-set torch function based on the model configuration file of the quantized hybrid expert model to determine the target memory environment; inputting the pre-running text dataset into the quantized hybrid expert model for pre-running inference in the target memory environment to determine the average computation time of the model; calculating the number of pre-fetchable experts using a pre-set expert prefetch inequality based on the average computation time of the model; and in the target memory environment, using the quantized hybrid expert model to infer the text data to be inferred based on the number of pre-fetchable experts, and outputting the model inference result.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a hybrid expert model reasoning method and apparatus. Background Technology

[0002] In recent years, with the rapid development of deep learning, large models have achieved remarkable results in various fields. To improve data privacy, the usability and personalization of AI functions, large models are increasingly being deployed to edge devices such as personal PCs, smartphones, and even the Internet of Things (IoT). For example, Qualcomm has deployed a text-based graph model containing over 1 billion parameters on a smartphone. Huawei has also embedded multimodal models in its smartphones to facilitate accurate content search based on natural language.

[0003] To meet the performance and power consumption requirements of edge devices, the industry considers 3B to be the golden size for edge devices. However, compared to dense models running 3B, sparse MoE (Mixture-of-Experts) models can provide better inference services with the same number of activated parameters. For example, the Qwen1.5-MoE model activates only 2.7B parameters, and its inference performance and speed are superior to various 7B models.

[0004] However, although the MoE model has a low number of activated parameters, its overall parameter count is relatively high. For example, Qwen1.5-MoE, despite activating only 2.7B parameters, has 14.3B parameters and requires approximately 30GB of memory to run inference. In contrast, edge devices often have much smaller memory footprints. For instance, NVIDIA's Jetson Orin Nano development board, a common device for embedded development, only has 8GB or 16GB of RAM. Most current smartphone models from major manufacturers also have 16GB or less of RAM.

[0005] Existing inference processes based on hybrid expert models mostly involve training a hash network (small model) to predict the expert that the input will select at each layer, thereby pre-fetching these experts to complete the inference. However, whenever a new model is introduced, additional training costs are required to train the corresponding small model for prediction, resulting in excessively high application costs. Summary of the Invention

[0006] This invention provides a hybrid expert model reasoning method and apparatus to solve the technical problem of low reasoning efficiency caused by existing hybrid expert model-based reasoning processes.

[0007] The first aspect of this invention provides a hybrid expert model reasoning method, comprising:

[0008] Obtain the pre-run text dataset and perform expert quantization on the pre-set hybrid expert model to determine the quantized hybrid expert model;

[0009] The initial memory environment corresponding to the quantized hybrid expert model is adjusted according to the model configuration file of the quantized hybrid expert model using a pre-set torch function to determine the target memory environment;

[0010] In the target memory environment, the pre-run text dataset is input into the quantized hybrid expert model for pre-run inference to determine the average computation time of the model;

[0011] The average duration is calculated based on the model using the pre-set expert prefetch inequality, and the number of experts that can be prefetched is calculated.

[0012] In the target memory environment, based on the number of prefetchable experts, the quantized hybrid expert model is used to infer the text data to be inferred, and the model inference result is output.

[0013] Optionally, the step of using a preset torch function to adjust the initial memory environment corresponding to the quantization hybrid expert model according to the model configuration file of the quantization hybrid expert model, and determining the target memory environment, includes:

[0014] Based on the preset torch function, the initial memory environment corresponding to the quantized hybrid expert model is identified, and the memory budget is output.

[0015] Based on the model configuration file, determine the shape and size of the network layers corresponding to the quantized hybrid expert model;

[0016] Calculate the memory size occupied by the network layer based on the shape and size of the network layer;

[0017] Subtract the memory budget from the memory size occupied by the network layer to output the remaining space budget;

[0018] The total number of experts that can be cached is determined by dividing the remaining space budget by the memory size of a single expert in the memory size occupied by the network layer.

[0019] Calculate the number of deep expert caches based on the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model.

[0020] Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, the initial memory environment corresponding to the quantized hybrid expert model is adjusted to determine the target memory environment.

[0021] Optionally, the quantized hybrid expert model includes an input embedding layer, multiple target hybrid expert modules, and an output embedding layer; the step of using the quantized hybrid expert model to perform inference on the text data to be inferred based on the number of prefetchable experts, and outputting the model inference result, includes:

[0022] The text data to be inferred is feature-mapped by the input embedding layer, and a text vector representation is output.

[0023] Based on the number of pre-fetchable experts, multiple target hybrid expert modules are used to perform hybrid expert calculations on the text vector representation to generate target intermediate states.

[0024] The target intermediate state is input into the output embedding layer for feature mapping, and the model inference result is output.

[0025] Optionally, the target hybrid expert module includes a gating layer, an expert layer, an attention layer, and a normalization layer; the data processing procedure for adjacent target hybrid expert modules includes:

[0026] The input vector data of the first target hybrid expert module is normalized through the normalization layer of the first target hybrid expert module to output a first initial normalized vector. The first initial normalized vector is then linearly transformed through the attention layer of the first target hybrid expert module to generate a first linear vector.

[0027] The first linear vector is normalized using the normalization layer of the first target hybrid expert module to generate a first intermediate normalized vector;

[0028] The first weight value corresponding to multiple experts in the expert layer of the first target hybrid expert module is determined by weight calculation based on the first intermediate normalized vector through the gating layer of the first target hybrid expert module.

[0029] Sort each of the first weight values ​​in descending order, select the expert corresponding to the first weight value of the number of experts that can be pre-selected as the first activated expert, and dequantize each of the first activated experts to determine multiple first dequantized activated experts.

[0030] Each of the first dequantization activation experts performs expert calculations on the first intermediate normalized vector, outputs the expert results corresponding to each of the first dequantization activation experts, and merges the expert results corresponding to each of the first dequantization activation experts to generate the first intermediate state.

[0031] The first intermediate state is normalized by the normalization layer of the second target hybrid expert module, and a second initial normalized vector is output. The second initial normalized vector is then linearly transformed by the attention layer of the second target hybrid expert module to generate a second linear vector.

[0032] The second linear vector is normalized using the normalization layer of the second target hybrid expert module to generate a second intermediate normalized vector;

[0033] The gating layer of the second target hybrid expert module is used to calculate the weight of the first intermediate normalized vector to determine the second weight value corresponding to multiple experts in the expert layer of the second target hybrid expert module.

[0034] Sort each of the second weight values ​​in descending order, select the expert corresponding to the second weight value of the number of experts that can be pre-selected as the second activated expert, and dequantize each of the second activated experts to determine multiple second dequantized activated experts.

[0035] Each of the second dequantization activation experts performs expert calculations on the second intermediate normalized vector, outputs the expert results corresponding to each of the second dequantization activation experts, and merges the expert results corresponding to each of the second dequantization activation experts to generate the second intermediate state.

[0036] Optionally, the average computation time of the model includes the computation time of a single expert, the average computation time of the attention layer, the average computation time of the gating layer, the average computation time of the expert layer, and the average computation time of the normalization layer; the preset expert prefetch inequality is specifically:

[0037]

[0038] Where x is the initial number of prefetchable experts; transmission duration 单个专家 For single expert transfer time, this represents the time required to load a single expert from disk into memory; computation time. 注意力层 Calculate the average duration for the attention layer; calculate duration 专家层 Calculate the average duration for the expert layer; calculation duration 归一化层 Calculate the average time for the normalized layer; calculation time 门控层 Calculate the average duration for the gated layer.

[0039] Optionally, the network layer data further includes the number of experts in the target hybrid expert module and the number of layers in the quantized hybrid expert model; the calculation process for the number of deep expert caches is as follows:

[0040] ;

[0041] Among them, the number of caches深层 Cache quantity for deep expert level; Cache quantity 总 The total number of cacheable experts; the number of fully cached layers is the number of shallow expert fully cached layers; the number of experts per layer is the number of experts in the target hybrid expert module; the total number of layers is the number of layers in the quantized hybrid expert model.

[0042] A second aspect of the present invention provides a hybrid expert model inference apparatus, comprising:

[0043] The acquisition module is used to acquire the pre-run text dataset and perform expert quantization on the pre-set hybrid expert model to determine the quantized hybrid expert model;

[0044] The adjustment module is used to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model using a preset torch function, and to determine the target memory environment;

[0045] The pre-running inference module is used to input the pre-running text dataset into the quantized hybrid expert model in the target memory environment for pre-running inference and to determine the average computation time of the model.

[0046] The calculation module is used to calculate the average duration based on the model using a preset expert prefetch inequality, and to calculate the number of experts that can be prefetched.

[0047] The output module is used to perform reasoning on the text data to be reasoned using the quantized hybrid expert model based on the number of prefetchable experts in the target memory environment, and output the model reasoning result.

[0048] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the hybrid expert model reasoning method as described in any of the preceding claims.

[0049] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the steps of the hybrid expert model reasoning method as described in any of the preceding claims.

[0050] The fifth aspect of the present invention provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein, when the program instructions are executed by a computer, the computer performs the steps of the hybrid expert model reasoning method as described in any of the preceding claims.

[0051] As can be seen from the above technical solutions, the present invention has the following advantages:

[0052] The above-mentioned technical solution of the present invention provides a hybrid expert model inference method. First, a pre-run text dataset is obtained, and the pre-set hybrid expert model is quantized to determine the quantized hybrid expert model. Next, a pre-set torch function is used to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model to determine the target memory environment. In the target memory environment, the pre-run text dataset is input into the quantized hybrid expert model for pre-run inference to determine the average computation time of the model. A pre-set expert prefetch inequality is used to calculate the number of prefetchable experts based on the average computation time of the model. Finally, in the target memory environment, based on the number of prefetchable experts, the quantized hybrid expert model is used to infer the text data to be inferred, and the model inference result is output. Based on the above scheme, the process of using a pre-set torch function to adjust the memory environment according to the model configuration file of the quantized hybrid expert model to obtain the target memory environment, calculating the number of prefetchable experts according to the pre-run inference process, and combining the target memory environment and the number of prefetchable experts to infer the text data to be inferred and outputting the model inference result eliminates the need to introduce additional training costs to train corresponding small models for prediction, thereby reducing application costs. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the steps of a hybrid expert model reasoning method provided in Embodiment 1 of the present invention.

[0055] Figure 2 This is a reasoning framework diagram of the hybrid expert model reasoning method provided in Embodiment 1 of the present invention;

[0056] Figure 3 This is a schematic diagram of the routing weights during prediction provided in Embodiment 1 of the present invention;

[0057] Figure 4 This is a schematic diagram illustrating the actual calculation of routing weights provided in Embodiment 1 of the present invention;

[0058] Figure 5 This is a schematic diagram illustrating the average prefetch accuracy of each layer provided in Embodiment 1 of the present invention;

[0059] Figure 6 This is a schematic diagram of the cross-layer expert prefetching process provided in Embodiment 1 of the present invention;

[0060] Figure 7 This is a comparative diagram of the traditional Transformer model and the MoE model provided in Embodiment 1 of the present invention;

[0061] Figure 8 This is a structural block diagram of a hybrid expert model inference device provided in Embodiment 2 of the present invention. Detailed Implementation

[0062] This invention provides a hybrid expert model reasoning method and apparatus to solve the technical problem of low reasoning efficiency caused by existing hybrid expert model-based reasoning processes.

[0063] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0064] Terminology Explanation:

[0065] Mixture-of-Experts (MoE) Models: MoE models replace the feedforward neural network layers in a Transformer with MoE layers. A MoE layer consists of a gating network and multiple experts. The gating function is a key feature of MoE; it discretely selects experts for each token decision. The most common approach is to select the top k experts for each token. In this case, the gating function is typically a softmax function, activating the top-k experts based on the calculated probability distribution. K is generally 1 or 2. The experts are essentially feedforward neural networks (FFNs).

[0066] Token: In natural language processing, a token typically refers to a meaningful piece of text. When large models process text, they break down the input sentence into individual tokens.

[0067] Pipeline: In computing, a pipeline breaks down a task into multiple independent stages, each of which can process different tasks in parallel. The I / O-computation pipeline discussed below primarily describes the parallel processing order and relationships between computation and data transfer.

[0068] B: Abbreviation for billion, a unit for the number of parameters in a large model. For example, 3B below means that the model has 3 billion parameters.

[0069] Model inference is divided into two stages: prefill and decoding.

[0070] Prefill stage: In this stage, the model receives a complete input sequence and performs computation all at once. Through this computation, the model generates all necessary intermediate states (such as key-value caches) and caches them, providing a foundation for subsequent stepwise generation. This is typically used for initial contextual understanding of sequences, especially for inference on long sequences.

[0071] Decoding Phase: In the decoding phase, the model uses the key-value cache stored in the pre-filling phase to generate tokens step by step. At each step, the model predicts the next token based on the output of the previous step and the cached state, and updates the key-value cache and other states until a complete output sequence is generated. This process is step-by-step and suitable for tasks such as dialogue generation and text continuation.

[0072] MMLU (Massive Multitask Language Understanding) is a large-scale benchmark dataset designed to evaluate the multitask learning capabilities of language models. It covers 57 tasks across a wide range of disciplines, including humanities, sciences, and mathematics, with difficulty levels ranging from high school to expert. This dataset aims to test a model's generalization ability across various tasks, particularly its ability to reason, apply knowledge, and solve multi-domain problems.

[0073] CUDA Stream: A concept in CUDA programming, referring to a queue of operations executed sequentially on the GPU. Each CUDA Stream can execute tasks independently, allowing different computation and data transfer operations to overlap, thereby achieving parallelization and higher resource utilization. By distributing tasks across multiple Streams, programs can improve performance and reduce latency, especially with significant optimization effects in scenarios involving a mix of computation and memory transfers.

[0074] Please see Figure 1 , Figure 1 The flowchart illustrates the steps of a hybrid expert model reasoning method provided in Embodiment 1 of the present invention.

[0075] This invention provides a hybrid expert model reasoning method, comprising:

[0076] Step 101: Obtain the pre-run text dataset and perform expert quantization on the pre-set hybrid expert model to determine the quantized hybrid expert model.

[0077] It should be noted that you should refer to [link / reference]. Figure 2The inference framework of this invention can be divided into two stages: the deployment stage and the inference stage. The deployment stage includes model quantization and expert caching (memory environment adjustment), while the inference stage includes expert management and pipeline management.

[0078] Furthermore, for model quantization, which is an essential part of edge inference, it can significantly reduce memory requirements and shorten I / O time. Since sparse activation experts account for the majority of parameters in the MoE model, and other parts besides the experts participate in the calculation in each round of inference, this invention chooses to store all parts other than the experts, namely attention layer, normalization layer, embedding layer, gating layer, etc., in memory with their original precision, and then only unload and quantize the experts.

[0079] Furthermore, after weighing the speed and accuracy of various quantization algorithms, this invention chooses to use semi-quadratic quantization to quantize the expert part. To select an appropriate number of quantization bits, this invention uses semi-quadratic quantization to quantize the Qwen-1.5-MoE model and performs inference using the MMLU dataset. The calculated accuracy loss is shown in Table 1. Clearly, INT4 quantization significantly reduces memory requirements while almost maintaining the original inference quality of the model.

[0080] Table 1. Impact of semi-quadratic quantization on model accuracy

[0081]

[0082] In this embodiment, the present invention uses semi-quadratic quantization to perform INT4 quantization on the expert part of the MoE model, while leaving the parts other than the experts unquantized and always maintaining them in memory. Furthermore, before each expert is used, the present invention uses a dequantization algorithm to convert it back to its original data type.

[0083] Step 102: Use the pre-set torch function to adjust the initial memory environment corresponding to the quantization hybrid expert model according to the model configuration file of the quantization hybrid expert model, and determine the target memory environment.

[0084] It's important to note that caching as many experts as possible in memory within existing memory constraints not only reduces I / O between memory and disk but also increases the probability of expert hits (a hit means the expert selected by the token is already in memory, eliminating the need for I / O). Furthermore, quantified experts have lower memory requirements, helping to cache more experts within limited memory space. Existing expert caching strategies often employ either Least Recently Used (LRU) or Least Frequently Used (LFU) strategies. These strategies respectively count the least recently used and least frequently used experts and evict them from memory. Therefore, this invention uses an Adaptive Replacement Cache (ARC) algorithm to dynamically balance LRU and LFU caching strategies. Specifically, this invention maintains two lists: one to track recently used cached items and the other to track frequently used cached items. ARC can automatically adjust the size of these two lists based on changes in access patterns, adapting to different workloads and providing a higher cache hit rate.

[0085] Further, step 102 may include the following sub-steps S21-S27:

[0086] S21. Based on the preset torch function, perform memory identification on the initial memory environment corresponding to the quantized hybrid expert model and output the memory budget;

[0087] S22. Based on the model configuration file, determine the shape and size of the network layers corresponding to the quantized hybrid expert model;

[0088] S23. Calculate the memory size occupied by the network layer based on its shape and size;

[0089] S24. Subtract the memory budget from the memory size occupied by the network layer, and output the remaining space budget.

[0090] S25. Divide the remaining space budget and the memory size of a single expert in the network layer memory size to determine the total number of experts that can be cached.

[0091] The network layer shape and size are the shape and size of each layer in the quantized hybrid expert model.

[0092] The memory usage of network layers includes the memory usage of a single expert, the memory usage of the attention layer, the memory usage of the normalization layer, the memory usage of the embedding layer, and the memory usage of the gating layer.

[0093] It should be noted that, in order to dynamically cache a large number of experts in different environments, this invention first uses functions in the torch library (pre-built torch functions) to identify the memory budget of the current environment. Then, using the model configuration file, the shape and size of each layer (network layer shape and size) are obtained to calculate the memory usage of a single expert, attention layer, normalization layer, embedding layer, and gating layer, i.e., the memory usage of the network layer. Next, the total memory usage of the attention layer, normalization layer, embedding layer, and gating layer is subtracted from the total memory budget to obtain the remaining space budget. Finally, the remaining space budget is divided by the memory usage of a single expert to determine the maximum number of experts that can be cached in the current environment, i.e., the total number of experts that can be cached.

[0094] S26. Calculate the number of deep expert caches based on the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model.

[0095] S27. Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, adjust the initial memory environment corresponding to the quantized hybrid expert model to determine the target memory environment.

[0096] The network layer data includes the number of shallow expert fully cached layers, the number of experts contained in the target hybrid expert module, and the number of layers in the quantized hybrid expert model.

[0097] It should be noted that the existing approach divides the total number of cacheable experts by the number of model layers to obtain the average number of cacheable experts per layer. However, this leads to lower accuracy for shallow expert prefetching and higher accuracy for deep expert prefetching. Therefore, to compensate for the low accuracy of shallow layers, this invention caches all experts in the shallow layers, and then distributes the remaining cacheable experts evenly among subsequent layers. The number of fully cached layers varies with the memory budget; assuming the quantized hybrid expert model has 24 layers, and all experts in the first three layers are considered shallow experts, then the number of fully cached shallow expert layers is 3.

[0098] Furthermore, the calculation process for the number of deep expert caches is as follows:

[0099] ;

[0100] Among them, the number of caches 深层 Cache quantity for deep expert level; Cache quantity 总 The total number of cacheable experts; the number of fully cached layers is the number of shallow expert fully cached layers; the number of experts per layer is the number of experts in the target hybrid expert module; the total number of layers is the number of layers in the quantized hybrid expert model.

[0101] It is worth mentioning that if the maximum number of experts that can be cached is insufficient to support multiple shallow layers of full caching, then the number of deep layers will be 0, and as many shallow layers as possible will be cached starting from the first layer.

[0102] Step 103: In the target memory environment, input the pre-run text dataset into the quantized hybrid expert model for pre-run inference and determine the average computation time of the model.

[0103] The average computation time of the model includes the average computation time of a single expert, the average computation time of the attention layer, the average computation time of the gating layer, the average computation time of the expert layer, and the average computation time of the normalization layer.

[0104] It should be noted that the specific number of experts prefetched, x, is difficult to determine because it is related to the computation time t between the two gating layers. If the total transmission time of x experts is much less than t, the opportunity to improve accuracy is wasted. If the total transmission time of x experts is greater than t, it will affect computational efficiency and increase inference latency. Therefore, in order to implement a pipeline that ensures complete overlap between computation and communication, this invention uses model-based pipeline management.

[0105] Furthermore, after the model deployment is completed, the present invention records the average computation time of the attention layer, gating layer, expert layer, normalization layer, etc., based on the pre-run inference process, and also records the time required to load a single expert from the hard disk into memory, i.e. the average computation time of the model.

[0106] Step 104: Calculate the average duration based on the model using the pre-set expert prefetch inequality, and calculate the number of experts that can be prefetched.

[0107] It should be noted that the pre-set expert prefetch inequality is as follows:

[0108]

[0109] Where x is the initial number of prefetchable experts; transmission duration 单个专家 For single expert transfer time, this represents the time required to load a single expert from disk into memory; computation time. 注意力层 Calculate the average duration for the attention layer; calculate duration 专家层 Calculate the average duration for the expert layer; calculation duration 归一化层 Calculate the average time for the normalized layer; calculation time 门控层 Calculate the average duration for the gated layer.

[0110] Furthermore, the initial number of prefetchable experts x calculated using the inequality is rounded down to obtain the final number of transmittable experts |x|, i.e., the number of prefetchable experts. Since this invention uses semi-quadratic quantization to quantize each expert as INT4 during the deployment phase, the transmission time is... 单个专家The significant reduction allows for a further increase in x. Secondly, this invention uses a caching algorithm, so not all |x| will require transmission, further facilitating the complete overlap of computation and communication.

[0111] Step 105: In the target memory environment, based on the number of prefetchable experts, use a quantized hybrid expert model to reason about the text data to be reasoned, and output the model reasoning results.

[0112] The quantized hybrid expert model consists of an input embedding layer, multiple objective hybrid expert modules, and an output embedding layer.

[0113] Specifically, step 105 may include the following sub-steps S51-S53:

[0114] Step S51: Perform feature mapping on the text data to be inferred through the input embedding layer, and output a text vector representation;

[0115] Step S52: Based on the number of prefetchable experts, multiple target hybrid expert modules are used to perform hybrid expert calculations on the text vector representation to generate target intermediate states;

[0116] Step S53: Input the target intermediate state into the output embedding layer for feature mapping, and output the model inference result.

[0117] It should be noted that after model deployment is complete, this invention dynamically manages experts upon receiving requests to achieve a fully overlapping pipeline, eliminating the impact of expert offloading on inference. In edge-side inference (MoE), the most critical issue is how to prefetch experts. Only with accurate expert prefetching can overlap between computation and communication be achieved, thereby reducing inference latency.

[0118] Furthermore, the present invention tested the cosine similarity between the inputs of the two gated layers and found that the cosine similarity was as high as 88.83%. Next, the inventors further analyzed the accuracy of this prefetching method and found that the accuracy reached 99.75% in the pre-filling stage and 78.79% in the decoding stage. This prefetching method achieves such high accuracy without introducing any additional costs, which is highly desirable.

[0119] Further, please refer to Figures 3-4 To further improve prefetching accuracy, this invention explores the routing weights calculated in the gating layer. During the gating layer's calculation process, a corresponding weight is assigned to each expert based on the input, and then the k experts with the highest weights are activated. For example... Figures 3-4 The figure shows the experimental results based on a gated layer in the Qwen-1.5-MoE model. The gated layers of the Qwen-1.5-MoE model are selected from 4 out of 60 experts. Figure 3The expert weights in the calculation are obtained through cross-layer computation. Based on this result, experts 5, 11, 21, and 36 should be pre-selected. However, the actual calculation result of this gating layer is as follows: Figure 4 As shown, it is clear that the experts who will actually be activated are 5, 21, 31, and 36. However, comparing the two figures ( Figure 3 and Figure 4 As can be seen, the distribution of route weights is extremely similar, and expert 31, which failed prefetching, is... Figure 3 It ranked 5th in the prediction results. This means that increasing the number of expert pre-selections can greatly improve the accuracy of expert pre-selection.

[0120] Further, please refer to Figure 5 Further experiments revealed that this approach achieves a prefetch accuracy of 97.15%. Therefore, in summary, this invention, while performing calculations at each gating layer, sends the input to the next gating layer, using the CPU for parallel cross-layer calculations. Then, based on the calculation results (i.e., routing weights), the top |x| experts are prefetched to achieve cost-free and highly accurate expert prefetching. The inventors discovered, while calculating the accuracy of prefetching at each layer, that… Figure 5 The phenomenon shown is that the accuracy of shallow expert prefetching is lower than that of deep prefetching.

[0121] For this purpose, please refer to Figure 6 This invention proposes cross-layer expert prefetching, which sends the input of the gating layer in the current MoE block (block i) to the gating layer in the next MoE block (block i+1) for computation. Since this computation process can be completed by the CPU and prefetching can begin immediately based on the computation result, the I / O process of this prefetching expert can overlap with the computation of the attention layer and expert layer between the two gating layers, thus avoiding any delay in the inference process.

[0122] Optionally, the target hybrid expert module includes a gating layer, an expert layer, an attention layer, and a normalization layer; the data processing of adjacent target hybrid expert modules can be achieved by executing the following steps S01 to S010:

[0123] Step S01: Normalize the input vector data of the first target hybrid expert module through the normalization layer of the first target hybrid expert module, output the first initial normalized vector, and perform a linear transformation on the first initial normalized vector through the attention layer of the first target hybrid expert module to generate the first linear vector.

[0124] Step S02: Normalize the first linear vector using the normalization layer of the first target hybrid expert module to generate the first intermediate normalized vector;

[0125] Step S03: The first weight value corresponding to multiple experts in the expert layer of the first target hybrid expert module is determined by calculating the weights based on the first intermediate normalized vector through the gate layer of the first target hybrid expert module.

[0126] Step S04: Sort each first weight value in descending order, select the expert corresponding to the first weight value of the number of experts that can be pre-selected as the first activated expert, and dequantize each first activated expert to determine multiple first dequantized activated experts.

[0127] Step S05: Each first dequantization activation expert performs expert calculations on the first intermediate normalized vector, outputs the expert results corresponding to each first dequantization activation expert, and merges the expert results corresponding to each first dequantization activation expert to generate the first intermediate state.

[0128] Step S06: Normalize the first intermediate state through the normalization layer of the second target hybrid expert module, output the second initial normalized vector, and perform a linear transformation on the second initial normalized vector through the attention layer of the second target hybrid expert module to generate the second linear vector;

[0129] Step S07: Normalize the second linear vector using the normalization layer of the second target hybrid expert module to generate the second intermediate normalized vector;

[0130] Step S08: Use the gating layer of the second target hybrid expert module to calculate the weight of the first intermediate normalized vector, and determine the second weight value corresponding to multiple experts in the expert layer of the second target hybrid expert module.

[0131] Step S09: Sort each second weight value in descending order, select the expert corresponding to the second weight value of the number of experts that can be pre-selected as the second activated expert, and dequantize each second activated expert to determine multiple second dequantized activated experts.

[0132] Step S010: Each second dequantization activation expert performs expert calculations on the second intermediate normalized vector, outputs the expert results corresponding to each second dequantization activation expert, and merges the expert results corresponding to each second dequantization activation expert to generate the second intermediate state.

[0133] The input vector data is the vector data input to the target hybrid expert module.

[0134] It should be noted that the gating layers in MoE module i (the first target hybrid expert module) and MoE module i+1 (the second target hybrid expert module) simultaneously receive and process the first intermediate normalized vector to obtain the first weight values ​​corresponding to multiple experts in the expert layer of the first target hybrid expert module and the second weight values ​​corresponding to multiple experts in the expert layer of the second target hybrid expert module. Based on each first weight value and each second weight value, the first and second activated experts for the pre-selected experts are determined. For example, assuming that the calculated number of pre-selected experts is 3, all the first weight values ​​and all the second weight values ​​are sorted in descending order, and the experts corresponding to the first 3 first weight values ​​are selected as the first activated experts, and the experts corresponding to the first 3 second weight values ​​are selected as the second activated experts.

[0135] Furthermore, after each expert completes its transmission, this invention uses another CUDA Stream to dequantize the expert in parallel as the original data type (such as BF16 / FP16) to reduce accuracy loss. The dequantization calculation for each expert is executed by the CPU, overlapping with model calculation and expert transmission, and thus does not introduce additional latency. That is, the activated expert is dequantized to obtain the dequantized expert (first dequantized activated expert and second dequantized activated expert).

[0136] Furthermore, each first dequantization activation expert performs expert calculations on the first intermediate normalized vector, outputting the expert results corresponding to each first dequantization activation expert. The expert results corresponding to each first dequantization activation expert are then merged to generate a first intermediate state. This first intermediate state is then processed sequentially through the normalization layer-attention layer-normalization layer in the second target hybrid expert module. After outputting the second intermediate normalized vector, each second dequantization activation expert performs expert calculations on the second intermediate normalized vector, outputting the expert results corresponding to each second dequantization activation expert. The expert results corresponding to each second dequantization activation expert are then merged to generate the second intermediate state.

[0137] Furthermore, if the second target hybrid expert module is not connected to the next target hybrid expert module, then the second intermediate state output by the second target hybrid expert module is taken as the target intermediate state; if the second target hybrid expert module is connected to the next target hybrid expert module, then the next target hybrid expert module connected to the second target hybrid expert module is taken as the new second target hybrid expert module, the second target hybrid expert module is taken as the new first target hybrid expert module, the second intermediate state is taken as the new first intermediate state, the second intermediate normalized vector is taken as the new first intermediate normalized vector, and then the process jumps to step S06 until the second target hybrid expert module is not connected to the next target hybrid expert module, and the second intermediate state output by the second target hybrid expert module is taken as the target intermediate state.

[0138] In this embodiment, the present invention achieves high-accuracy expert prefetching through a simple and cost-free method (cross-layer prefetching), and further improves prefetching accuracy by utilizing routing weights. Furthermore, the present invention uses quantization and pipelined modeling to achieve complete overlap between communication and computation, virtually eliminating inference latency caused by model offloading. Simultaneously, the present invention provides higher-quality large-model inference services by deploying MoE models on the edge, realizing a MoE inference framework for fast inference on the edge.

[0139] As a comparison of technical effects, existing technologies can be used as a reference. For the memory optimization problem in the inference process of large models, unloading is one of the current mainstream solutions. Tensors that are not currently involved in the calculation can be unloaded to the hard disk and loaded into memory when needed, so as to greatly reduce the memory requirements. However, due to the physical limitation that the current calculation speed is generally faster than the transmission speed, unloading is often accompanied by serious IO overhead. In this regard, dense models can often prefetch the parameters of the next layer in advance to achieve the overlap of calculation and IO. However, the MoE model is difficult to accurately prefetch due to the dynamic activation of experts, and the IO overhead of loading the entire MoE layer is large, which will lead to high latency. There are two main solutions to this problem: (1) Activation-aware method. Based on the expert selection of the token in the previous or multiple layers, predict the expert selection of the token in the next layer based on statistical data. (2) Prediction model. Train a small network to predict the expert selection of the token in each layer. However, the prediction accuracy of method (1) is not satisfactory, and method (2) will introduce additional overhead and has poor portability. A corresponding small model must be retrained for each new MoE model.

[0140] Further, please refer to Figure 7 Existing large models are mainly based on the Transformer structure, such as Figure 7As shown in (a). Since GShard introduced the novel hybrid expert structure, researchers have continuously explored its enormous potential in improving the performance of large language models, and it has gradually become one of the mainstream structures for large language models. Popular models such as GPT-4, Gemini 1.5, and Mixtral 8x7B all use the MoE structure. Unlike large models based on the Transformer structure, hybrid expert models use MoE layers to replace feedforward neural network layers, such as... Figure 7 As shown in (b), the MoE layer consists of a gating network and multiple experts. The gating function is a key feature of MoE, discretely selecting experts to participate in the computation for each input. The most common approach is to select the top k experts for each token. In this case, the gating function is usually a softmax function, activating the top-k experts according to the calculated probability distribution, where k is typically 1 or 2. Current methods struggle to accurately predict expert selection.

[0141] Experts are essentially feedforward neural networks, sparsely activated based on the results of a gating function. It is this sparse activation characteristic that makes it the most feasible method for obtaining larger-scale models. For each input, the final output of the MoE layer is a weighted sum of the outputs from the selected experts.

[0142] To address memory optimization issues during large model inference, offloading is currently one of the mainstream solutions. Edge devices typically only have two levels of storage: RAM and hard disk. During model inference, since tensors are not involved in the entire computation, offloading tensors that are not currently involved in computation to the hard disk can free up a significant amount of memory. Therefore, theoretically, as long as there is enough memory to accommodate computational resources and enough hard disk to offload other resources, model inference can be performed on the edge. However, offloading strategies always come with latency issues. Because computation requires relevant tensors to reside in memory, the speed of tensor transfer between memory and hard disk is often slower than the SoC's computation speed due to bandwidth limitations. The SoC then has to wait for the tensor transfer to complete before performing computation, resulting in wasted computing power.

[0143] Quantization techniques reduce the model's storage and computational requirements by converting high-precision parameters (such as 32 / 16-bit floating-point numbers, FP32 / FP16 / BF16, etc.) into low-precision representations (such as 8 bits or lower, INT8 / INT4 / INT2 / INT1, etc.). This significantly reduces model size and computational complexity while maintaining relatively stable model performance, and is typically used to accelerate model inference and for deployment on resource-constrained devices (such as edge mobile devices).

[0144] In summary, existing inference methods primarily quantize experts by different numbers of bits based on their importance, thereby caching more experts in memory and reducing parameter transmission time. This method predicts experts through activation perception, asynchronously transmitting and dequantizing the predicted experts each time attention layer computation is performed. Since the experts are already quantized, transmission time is significantly reduced, making it easier to overlap computation and communication, thus lowering inference latency. Simultaneously, a hash network (small model) is trained to predict the experts selected at each layer, pre-fetching these experts to achieve low-latency MoE model inference in resource-constrained environments. Furthermore, the original model structure is modified. The MoE model selects the activated experts at each layer through gating, but the gating function for that layer is removed, and a pre-gate is added to the previous layer, retraining these gating networks. Therefore, during inference, the experts to be activated at layer i+1 can be calculated at layer i, and then asynchronous expert preloading is performed, overlapping the computation of layer i with the expert loading of layer i+1, further reducing inference latency.

[0145] Therefore, it can be seen that existing inference methods are based on activation-based perceptual prediction experts. Experiments have shown that the accuracy of prefetching is unsatisfactory, which seriously affects inference latency. Moreover, they mainly consider improving inference speed in the decoding stage, neglecting the more complex prefill stage; furthermore, they rely on a newly trained small network. While this approach based on a small model prediction expert achieves higher accuracy, it introduces additional training costs. Furthermore, this approach has very poor portability; whenever a new model is released, a corresponding small model must be trained for prediction to apply this technique, which is impractical. In addition, Scheme 3 alters the original structure of the model.

[0146] To address the aforementioned issues, this invention proposes a hybrid expert model inference method to achieve high-throughput MoE inference in resource-constrained scenarios. Specifically, this invention first analyzes the reasons for the inefficiency of current MoE inference pipelines, and then proposes an expert-aware multi-batch processing pipeline. This pipeline leverages the high computational and low I / O characteristics of popular experts to orchestrate multi-batch computations, minimizing inter-layer and intra-layer bubbles in the pipeline. Furthermore, this invention adaptively measures the hardware capabilities of the current environment to generate the optimal execution plan for that environment. Simultaneously, this invention focuses on the inference speed at the MoE endpoint. Compared to existing technologies, the expert prefetching accuracy of this invention is significantly higher, and its implementation is simpler. Moreover, this invention is less expensive, requiring no additional training costs. The prefetching method of this invention can be directly used in the inference process of any MoE model, exhibiting far superior technology portability compared to existing technologies. In summary, this invention achieves high-accuracy expert prefetching in a simple and efficient manner, significantly reducing the inference latency caused by expert offloading, and enabling fast inference of MoE models at the endpoint.

[0147] In this embodiment of the invention, a hybrid expert model inference method is provided. First, a pre-run text dataset is acquired, and the pre-set hybrid expert model is quantized to determine the quantized hybrid expert model. Next, a pre-set torch function is used to adjust the initial memory environment corresponding to the quantized hybrid expert model based on the model configuration file, determining the target memory environment. In the target memory environment, the pre-run text dataset is input into the quantized hybrid expert model for pre-run inference, determining the average computation time of the model. A pre-set expert prefetch inequality is used to calculate the number of prefetchable experts based on the average computation time of the model. Finally, in the target memory environment, based on the number of prefetchable experts, the quantized hybrid expert model is used to infer the text data to be inferred, outputting the model inference result. Based on the above scheme, the process of using a pre-set torch function to adjust the memory environment according to the model configuration file of the quantized hybrid expert model to obtain the target memory environment, calculating the number of prefetchable experts based on the pre-run inference process, and combining the target memory environment and the number of prefetchable experts to infer the text data to be inferred and outputting the model inference result eliminates the need to introduce additional training costs to train corresponding small models for prediction, thereby reducing application costs.

[0148] Please see Figure 8 , Figure 8 This is a structural block diagram of a hybrid expert model inference device provided in Embodiment 2 of the present invention.

[0149] The present invention provides a hybrid expert model inference device, comprising:

[0150] The acquisition module 801 is used to acquire the pre-run text dataset and perform expert quantization on the pre-set hybrid expert model to determine the quantized hybrid expert model;

[0151] The adjustment module 802 is used to adjust the initial memory environment corresponding to the quantization hybrid expert model according to the model configuration file of the quantization hybrid expert model using a preset torch function, and to determine the target memory environment;

[0152] The pre-running inference module 803 is used to input the pre-running text dataset into the quantized hybrid expert model for pre-running inference in the target memory environment and determine the average computation time of the model.

[0153] Calculation module 804 is used to calculate the average duration based on the model using a preset expert prefetch inequality and to calculate the number of experts that can be prefetched.

[0154] The output module 805 is used to perform reasoning on the text data to be reasoned in the target memory environment based on the number of prefetchable experts and using a quantized hybrid expert model, and output the model reasoning results.

[0155] Furthermore, module 802 is adjusted specifically for:

[0156] Based on the pre-set torch function, memory identification is performed on the initial memory environment corresponding to the quantized hybrid expert model, and the memory budget is output.

[0157] Based on the model configuration file, determine the shape and size of the network layers corresponding to the quantized hybrid expert model;

[0158] Calculate the memory usage of each network layer based on its shape and size;

[0159] Subtract the memory budget from the memory usage of the network layer and output the remaining space budget.

[0160] Divide the remaining space budget and the memory size of a single expert in the network layer memory size to determine the total number of experts that can be cached.

[0161] The number of deep experts to be cached is calculated based on the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model.

[0162] Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, the initial memory environment corresponding to the quantized hybrid expert model is adjusted to determine the target memory environment.

[0163] Furthermore, the quantized hybrid expert model includes an input embedding layer, multiple target hybrid expert modules, and an output embedding layer; the output module 805 is specifically used for:

[0164] The text data to be inferred is feature-mapped by the input embedding layer, and the output is a text vector representation.

[0165] Based on the number of pre-fetchable experts, multiple target hybrid expert modules are used to perform hybrid expert calculations on the text vector representation to generate target intermediate states.

[0166] The intermediate state of the target is input into the output embedding layer for feature mapping, and the model inference result is output.

[0167] Optionally, the target hybrid expert module includes a gating layer, an expert layer, an attention layer, and a normalization layer; the data processing procedure for adjacent target hybrid expert modules includes:

[0168] The input vector data of the first target hybrid expert module is normalized through the normalization layer of the first target hybrid expert module, and the first initial normalized vector is output. The first initial normalized vector is then linearly transformed through the attention layer of the first target hybrid expert module to generate the first linear vector.

[0169] The first linear vector is normalized using the normalization layer of the first target hybrid expert module to generate the first intermediate normalized vector;

[0170] The first weight value of multiple experts in the expert layer of the first target hybrid expert module is determined by weight calculation based on the first intermediate normalized vector through the gate layer of the first target hybrid expert module.

[0171] Sort each first weight value in descending order, select the expert corresponding to the first weight value of the number of experts that can be pre-selected as the first activated expert, and dequantize each first activated expert to determine multiple first dequantized activated experts.

[0172] Each first dequantization activation expert performs expert calculations on the first intermediate normalized vector, outputs the expert results corresponding to each first dequantization activation expert, and merges the expert results corresponding to each first dequantization activation expert to generate the first intermediate state.

[0173] The first intermediate state is normalized by the normalization layer of the second target hybrid expert module, and the second initial normalized vector is output. The second initial normalized vector is then linearly transformed by the attention layer of the second target hybrid expert module to generate the second linear vector.

[0174] The second linear vector is normalized using the normalization layer of the second objective hybrid expert module to generate the second intermediate normalized vector;

[0175] The gating layer of the second target hybrid expert module is used to calculate the weight of the first intermediate normalized vector, and the second weight value corresponding to multiple experts in the expert layer of the second target hybrid expert module is determined.

[0176] Sort each second weight value in descending order, select the expert corresponding to the second weight value of the number of experts that can be pre-selected as the second activated expert, and dequantize each second activated expert to determine multiple second dequantized activated experts.

[0177] Each second dequantization activation expert performs expert calculations on the second intermediate normalized vector, outputs the expert results corresponding to each second dequantization activation expert, and merges the expert results corresponding to each second dequantization activation expert to generate the second intermediate state.

[0178] Furthermore, the average computation time of the model includes the computation time of a single expert, the average computation time of the attention layer, the average computation time of the gating layer, the average computation time of the expert layer, and the average computation time of the normalization layer; a pre-defined expert prefetch inequality is provided, specifically:

[0179]

[0180] Where x is the initial number of prefetchable experts; transmission duration 单个专家 For single expert transfer time, this represents the time required to load a single expert from disk into memory; computation time. 注意力层 Calculate the average duration for the attention layer; calculate duration 专家层 Calculate the average duration for the expert layer; calculation duration 归一化层 Calculate the average time for the normalized layer; calculation time 门控层 Calculate the average duration for the gated layer.

[0181] Furthermore, the network layer data also includes the number of experts in the target hybrid expert module, the number of layers in the quantized hybrid expert model, and the calculation process for the number of deep expert caches, specifically:

[0182] ;

[0183] Among them, the number of caches 深层 Cache quantity for deep expert level; Cache quantity 总 The total number of cacheable experts; the number of fully cached layers is the number of shallow expert fully cached layers; the number of experts per layer is the number of experts in the target hybrid expert module; the total number of layers is the number of layers in the quantized hybrid expert model.

[0184] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0185] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the hybrid expert model reasoning method as described in any of the above embodiments.

[0186] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implement the steps of the hybrid expert model reasoning method as described in any of the above embodiments.

[0187] This invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the hybrid expert model reasoning method as described in any of the above embodiments.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hybrid expert model reasoning method, characterized in that, include: Obtain the pre-run text dataset and perform expert quantization on the pre-set hybrid expert model to determine the quantized hybrid expert model; The initial memory environment corresponding to the quantized hybrid expert model is adjusted using a pre-set torch function based on the model configuration file of the quantized hybrid expert model, and the target memory environment is determined, including: Based on the preset torch function, the initial memory environment corresponding to the quantized hybrid expert model is identified, and the memory budget is output. Based on the model configuration file, determine the shape and size of the network layers corresponding to the quantized hybrid expert model; Calculate the memory size occupied by the network layer based on the shape and size of the network layer; Subtract the memory budget from the memory size occupied by the network layer to output the remaining space budget; The total number of experts that can be cached is determined by dividing the remaining space budget by the memory size of a single expert in the memory size occupied by the network layer. Calculate the number of deep expert caches based on the total number of cacheable experts and the network layer data corresponding to the quantized hybrid expert model. Based on the number of deep expert caches and the number of shallow expert full cache layers in the network layer data, the initial memory environment corresponding to the quantized hybrid expert model is adjusted to determine the target memory environment; In the target memory environment, the pre-run text dataset is input into the quantized hybrid expert model for pre-run inference to determine the average computation time of the model; The average duration is calculated based on the model using the pre-set expert prefetch inequality, and the number of experts that can be prefetched is calculated. In the target memory environment, based on the number of prefetchable experts, the quantized hybrid expert model is used to infer the text data to be inferred, and the model inference result is output.

2. The hybrid expert model reasoning method according to claim 1, characterized in that, The quantized hybrid expert model includes an input embedding layer, multiple target hybrid expert modules, and an output embedding layer; Based on the number of pre-fetchable experts, the quantized hybrid expert model is used to infer the text data to be inferred, and the model inference results are output, including: The text data to be inferred is feature-mapped by the input embedding layer, and a text vector representation is output. Based on the number of pre-fetchable experts, multiple target hybrid expert modules are used to perform hybrid expert calculations on the text vector representation to generate target intermediate states. The target intermediate state is input into the output embedding layer for feature mapping, and the model inference result is output.

3. The hybrid expert model reasoning method according to claim 2, characterized in that, The target hybrid expert module includes a gating layer, an expert layer, an attention layer, and a normalization layer; the data processing procedure for adjacent target hybrid expert modules includes: The input vector data of the first target hybrid expert module is normalized through the normalization layer of the first target hybrid expert module to output a first initial normalized vector. The first initial normalized vector is then linearly transformed through the attention layer of the first target hybrid expert module to generate a first linear vector. The first linear vector is normalized using the normalization layer of the first target hybrid expert module to generate a first intermediate normalized vector; The first weight value corresponding to multiple experts in the expert layer of the first target hybrid expert module is determined by weight calculation based on the first intermediate normalized vector through the gating layer of the first target hybrid expert module. Sort each of the first weight values ​​in descending order, select the expert corresponding to the first weight value of the number of experts that can be pre-selected as the first activated expert, and dequantize each of the first activated experts to determine multiple first dequantized activated experts. Each of the first dequantization activation experts performs expert calculations on the first intermediate normalized vector, outputs the expert results corresponding to each of the first dequantization activation experts, and merges the expert results corresponding to each of the first dequantization activation experts to generate the first intermediate state. The first intermediate state is normalized by the normalization layer of the second target hybrid expert module, and a second initial normalized vector is output. The second initial normalized vector is then linearly transformed by the attention layer of the second target hybrid expert module to generate a second linear vector. The second linear vector is normalized using the normalization layer of the second target hybrid expert module to generate a second intermediate normalized vector; The gating layer of the second target hybrid expert module is used to calculate the weight of the first intermediate normalized vector to determine the second weight value corresponding to multiple experts in the expert layer of the second target hybrid expert module. Sort each of the second weight values ​​in descending order, select the expert corresponding to the second weight value of the number of experts that can be pre-selected as the second activated expert, and dequantize each of the second activated experts to determine multiple second dequantized activated experts. Each of the second dequantization activation experts performs expert calculations on the second intermediate normalized vector, outputs the expert results corresponding to each of the second dequantization activation experts, and merges the expert results corresponding to each of the second dequantization activation experts to generate the second intermediate state.

4. The hybrid expert model reasoning method according to claim 1, characterized in that, The model's average computation time includes the average computation time of a single expert transmission, the average computation time of the attention layer, the average computation time of the gating layer, the average computation time of the expert layer, and the average computation time of the normalization layer; the pre-set expert prefetch inequality is specifically: ； Where x is the initial number of prefetchable experts; transmission duration 单个专家 For single expert transfer time, this represents the time required to load a single expert from disk into memory; computation time. 注意力层 Calculate the average duration for the attention layer; calculate duration 专家层 Calculate the average duration for the expert layer; calculation duration 归一化层 Calculate the average time for the normalized layer; calculation time 门控层 Calculate the average duration for the gated layer.

5. The hybrid expert model reasoning method according to claim 1, characterized in that, The network layer data also includes the number of experts in the target hybrid expert module and the number of layers in the quantized hybrid expert model; the calculation process for the number of deep expert caches is as follows: ； Among them, the number of caches 深层 Cache quantity for deep expert level; Cache quantity 总 The total number of cacheable experts; the number of fully cached layers is the number of shallow expert fully cached layers; the number of experts per layer is the number of experts in the target hybrid expert module; the total number of layers is the number of layers in the quantized hybrid expert model.

6. A hybrid expert model reasoning apparatus, applied to the hybrid expert model reasoning method of claim 1, characterized in that, include: The acquisition module is used to acquire the pre-run text dataset and perform expert quantization on the pre-set hybrid expert model to determine the quantized hybrid expert model; The adjustment module is used to adjust the initial memory environment corresponding to the quantized hybrid expert model according to the model configuration file of the quantized hybrid expert model using a preset torch function, and to determine the target memory environment; The pre-running inference module is used to input the pre-running text dataset into the quantized hybrid expert model in the target memory environment for pre-running inference and to determine the average computation time of the model. The calculation module is used to calculate the average duration based on the model using a preset expert prefetch inequality, and to calculate the number of experts that can be prefetched. The output module is used to perform reasoning on the text data to be reasoned using the quantized hybrid expert model based on the number of prefetchable experts in the target memory environment, and output the model reasoning result.

7. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the hybrid expert model reasoning method as described in any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the hybrid expert model reasoning method as described in any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, wherein when the program instructions are executed by a computer, the computer performs the hybrid expert model reasoning method as described in any one of claims 1-5.

Images