A MoE Batch Inference Method and System Based on Priority Differential Caching

Abstract

This invention belongs to the field of computer and artificial intelligence technology, and discloses a MoE batch inference method and system based on priority differential caching. The method includes: identifying globally active experts with high activation frequencies at each layer and resident them in a high-priority cache during initialization, establishing an expert priority list for each layer; selecting the set of experts to be activated in the current layer during the inference phase; dynamically updating expert priority scores based on expert activation and reuse behavior; migrating and replacing experts among high, medium, and low-level caches based on priority scores and cache thresholds; predicting the set of experts that may be activated in the next layer based on the current layer gating results and historical activation trajectories, and asynchronously loading them into a low-priority cache during the current layer's computation to achieve time overlap between computation and communication. This invention reduces the number of expert migrations, improves cache hit rate and throughput, significantly optimizes memory utilization, and has good scalability and versatility.

Description

Technical Field

[0001] This invention belongs to the field of computer and artificial intelligence technology, and more specifically, relates to a MoE batch inference method and system based on priority differential caching. Background Technology

[0002] In recent years, the rapid development of large-scale language models has driven the continuous expansion of the Transformer architecture, with its model size and number of parameters growing exponentially, resulting in a sharp increase in computational and storage overhead. To reduce computational costs while maintaining model capabilities, Mixture of Experts (MoE) models have gradually become an important architecture for large-scale model inference.

[0003] The MoE model is structurally based on sparse activation. Its core idea is to replace the traditional feedforward network with a MoE layer containing multiple experts. The router selects some experts to participate in the computation based on input features, thus reducing the actual computational cost while maintaining model capacity. Each input activates only a small number of experts, enabling the model to achieve stronger expressive power with the same computational budget. However, this sparse structure also leads to the problem of storing and transferring massive amounts of expert parameters. In mainstream MoE models, the expert part typically accounts for more than 95% of the total parameters. When the model is deployed on a single GPU or in a resource-constrained environment, the expert parameters cannot be fully resident in GPU memory, becoming a major bottleneck in the inference process.

[0004] To address the issue of insufficient GPU memory, existing MoE inference solutions typically offload expert parameters to host memory and load currently active experts onto the GPU on demand during inference. However, limited by the finite PCIe bandwidth between the GPU and the host, frequent expert migrations cause significant communication latency. As the inference batch size increases, the number of expert loading and unloading operations increases dramatically, making communication the dominant bottleneck for system performance and significantly limiting end-to-end throughput.

[0005] To alleviate these problems, existing research mainly optimizes the process through two approaches: one is a prefetch-based expert loading scheme, which predicts and loads experts that may be activated in the next layer during the current layer's computation, aiming to overlap computation and communication; the other is a caching-based expert reuse scheme, which caches some frequently activated experts on the GPU to reduce the number of data migrations from host to device. However, these schemes are generally optimized for single-batch inference scenarios. When extended to multi-batch inference, new bottlenecks emerge in communication and cache management: for the prefetch scheme, as the batch size increases, the number of experts that need to be migrated in each round of inference increases synchronously, and the communication time and computation time cannot be completely overlapped, resulting in a sharp increase in communication latency; for the caching scheme, when the batch size exceeds the cache capacity, the expert cache is frequently replaced, and the hit rate drops rapidly. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a MoE batch inference method and system based on priority differential caching, aiming to improve the throughput and cache hit rate of large model inference in a host-GPU heterogeneous environment.

[0007] To achieve the above objectives, this invention provides a MoE batch inference method based on priority differential caching, comprising: S1: Based on the activation frequency of experts in each layer of the MoE model, identify the global hot experts for the corresponding layer, and establish an independent priority list for experts in each layer. The priority list records the priority score of the experts in that layer. The priority score of the global hot experts is the highest score, and the initial priority score of the other experts in that layer is the lowest score. In the GPU, high and medium priority caches are configured independently for each layer, and a low priority cache is configured for all layers. The high priority cache is used to store the parameters of the global hot experts. S2: In the current round of the inference phase, the routing gating network in the MoE model selects the set of experts to be activated in the current layer based on the characteristics of the multiple batches of inference inputs in the current layer. S3: If the expert of the current layer is in the expert set, then increase the priority score of the expert; otherwise, decrease the priority score of the expert to obtain the updated priority score of the expert of the current layer. S4: Determine whether the expert parameter in the expert set can be found in any of the high, medium, and low priority caches. If yes, the expert is loaded. If not, the expert parameter is loaded from the CPU into the active expert cache area of ​​the current layer in the low priority cache, thus completing the expert loading. After all experts in the expert set are loaded, a parallel inference calculation of the features of the current layer's multi-batch inference input and all expert parameters of the current layer is triggered to obtain the current layer's output result, thereby obtaining the features of the next layer's multi-batch inference input. When the expert priority score in the active expert cache area of ​​the current layer is higher than the preset cache threshold, if the medium priority cache is not full, the expert parameters with high priority scores in the low priority cache are migrated to the medium priority cache. If the medium priority cache is full, the expert parameters with high priority scores in the low priority cache are used to replace the expert parameters in the medium priority cache with priority scores lower than the preset cache threshold according to their priority scores from low to high. Then, the active expert cache area of ​​the current layer is cleared. S5: Predict the set of experts to be activated in the next layer and load it into the next layer prediction expert cache in the low-priority cache; take the next layer as the current layer and return to S2 for iteration until all layers of the MoE model have been iterated, and take the output of the last layer as the inference result of the current round.

[0008] Furthermore, in S3, reducing the expert's priority score includes: If the expert parameter is located in the medium-priority cache, reduce the expert's priority score by one point; If the expert parameter is not in the medium priority cache, reduce the expert's priority score by a second score, where the second score is less than the first score.

[0009] Furthermore, S2 includes: The routing gating network in the MoE model calculates the activation probability of each expert required for each task in the current layer based on the characteristics of multiple batches of input in the current layer. The set of the k experts with the highest activation probabilities for all tasks in the current layer's multiple batches of inference is taken as the expert set.

[0010] Furthermore, in S1, the global popular experts of the corresponding layer are identified based on the expert activation frequency of each layer of the MoE model, including: in the fine-tuning stage before inference, the expert activation frequency in each layer of the MoE model is counted, and experts whose expert activation frequency is higher than a preset threshold are identified as global popular experts of the corresponding layer.

[0011] Furthermore, predict the set of experts that need to be activated in the next layer, including: The experts to be activated for each task in the current layer of the expert set to be activated, as well as the historical expert activation distribution data for that task, are input into the GRU-based predictor to obtain the next layer expert activation probability distribution for that task. The expert activation probability distributions of each task in the current layer multi-batch inference are merged to obtain the set of expert activation probability distributions in the next layer multi-batch inference. Based on the batch size, select the top N experts with the highest expert activation probability distribution from the expert activation probability distribution set as the next layer of experts to be activated in the prediction.

[0012] Furthermore, the capacity of the medium-priority cache is twice that of the high-priority cache.

[0013] The present invention also provides a MoE batch inference system based on priority differential caching, including a GPU, wherein the GPU includes: a preprocessing module, a priority management module, a differential caching module and a prediction prefetching module; The preprocessing module is used to identify the global hot experts for each layer based on the activation frequency of experts in each layer of the MoE model, and to establish an independent priority list for experts in each layer. The priority list records the priority score of the experts in that layer, with the priority score of the global hot experts being the highest score, and the initial priority scores of the other experts in that layer being the lowest scores. It is also used to configure high and medium priority caches independently for each layer, and to configure a shared low priority cache for all layers. The high priority cache is used to store the parameters of the global hot experts. In the current round of the inference phase, the routing gating network in the MoE model selects the set of experts to be activated in the current layer based on the characteristics of the multiple batches of inference inputs in the current layer. The priority management module is used to update the expert priority score of the current layer, including: if the expert of the current layer is in the expert set, then increase the priority score of the expert; otherwise, decrease the priority score of the expert, so as to obtain the updated expert priority score of the current layer. The differential caching module is used to determine whether the expert parameter in the expert set can be found in any of the high, medium, and low priority caches. If so, the expert is loaded. If not, the expert parameter is loaded from the CPU into the current layer's active expert cache area in the low priority cache, thus completing the expert loading. After all experts in the expert set are loaded, a parallel inference calculation of the features of the current layer's multi-batch inference input and all expert parameters of the current layer is triggered to obtain the current layer's output result, thereby obtaining the features of the next layer's multi-batch inference input. It is also used to, when the expert priority score in the current layer's active expert cache area is higher than a preset cache threshold, if the medium priority cache is not full, migrate the expert parameters with high priority scores in the low priority cache to the medium priority cache. If the medium priority cache is full, use the expert parameters with high priority scores in the low priority cache to replace the expert parameters in the medium priority cache with expert priority scores lower than the preset cache threshold according to their priority scores from low to high. Then, the current layer's active expert cache area is cleared. The prediction prefetching module is used to predict the set of experts to be activated in the next layer and load it into the next layer prediction expert cache area in the low priority cache. The next layer is used as the current layer, and the routing gating network, priority management module, differential cache module and prediction prefetching module are repeatedly executed until all layers of the MoE model are iterated. The output of the last layer is used as the inference result of the current round.

[0014] The present invention also provides a MoE batch inference processor based on priority differential caching, including a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the MoE batch inference method based on priority differential caching described above.

[0015] The present invention also provides a computer program product, including a computer program that, when the computer program is run on a computer, causes the computer to execute the MoE batch inference method based on priority differential caching as described above.

[0016] The present invention also provides a MoE batch inference collaborative processing system based on priority differential caching, comprising: CPU and GPU; The CPU is used to store expert parameters for each layer of the MoE model; The GPU is used to execute the MoE batch inference method based on priority differential caching as described in any one of claims 1-6.

[0017] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) In view of the significant communication and caching bottlenecks in existing MoE inference schemes in multi-batch scenarios, this invention provides a mechanism that can perform hierarchical management and dynamic priority scheduling based on expert access characteristics. In a CPU-GPU heterogeneous environment, a three-level priority differential cache is set in the GPU, that is, each layer of the MoE model is independently configured with high and medium priority caches, and a low priority cache is configured for all layers. The high priority cache resides in the global popular experts of each layer for a long time and sets their priority score to the highest score to avoid repeated migration during inference. The medium priority cache is used to retain the local popular experts of each layer for a short time. The active expert cache area in the low priority cache is used to temporarily store the active experts of the current layer. The hot and cold differentiation of experts is achieved by increasing or decreasing the priority score of experts in each layer, that is, the hierarchical migration and replacement of expert parameters in the priority differential cache. This invention employs a priority-driven hierarchical caching structure, periodically performing priority adjustments and cache migrations. This allows high-frequency experts to reside in a high-priority cache, mid-frequency experts to be retained in a mid-priority cache, and low-frequency experts to be loaded into a low-priority cache on demand. In multi-batch scenarios, this reduces the number of expert migrations from the host to the GPU, significantly improving expert cache hit rate and GPU memory utilization efficiency. During the computation of the current layer, by predicting and prefetching the parameters of subsequent layer experts into the next layer's prediction expert cache in the low-priority cache, efficient overlap between communication and computation is achieved, reducing communication latency. This invention's solution makes no intrusive changes to the existing MoE model structure, possesses good scalability and versatility, and can be widely applied to accelerate batch inference for large-scale language models and multimodal models.

[0018] (2) As a preferred option, the capacity of the medium-priority cache is twice that of the high-priority cache, which can achieve a balance between fixed storage (high-priority cache) and dynamic changes (medium-priority cache).

[0019] In summary, the MoE batch inference method based on priority differential caching of the present invention is applicable to hybrid expert (MoE) large model inference services in host-GPU heterogeneous environments. It realizes hierarchical scheduling and prediction prefetching of expert cache, which significantly alleviates communication pressure and improves the overall system performance. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the MoE batch inference system based on priority differential caching in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the MoE batch inference method based on priority differential caching in an embodiment of the present invention. Figure 3 This is a schematic diagram of the host-GPU heterogeneous parameter storage and migration structure in an embodiment of the present invention; Figure 4 This is a schematic diagram of the hierarchical structure of the differential cache module in an embodiment of the present invention; Figure 5 This is a structural diagram of the MoE model; Figure 6 This is a schematic diagram of a priority-driven cache update example in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] Example 1 like Figure 1As shown, this invention provides a MoE batch inference system based on priority differential caching, applied to accelerate large-scale language model inference in a host-GPU heterogeneous environment. The system includes a host and a GPU. The host is responsible for storing all expert parameters in the MoE model and scheduling and loading requests for expert parameters required by the GPU. The GPU side includes a differential caching module and a MoE processing unit. The differential caching module is used for hierarchical management of expert parameters, i.e., hierarchical scheduling of differential caches (high, medium, and low priority caches). The MoE processing unit includes a preprocessing module, a routing gating network in the MoE model, a priority management module, and a prediction prefetching module. The preprocessing module is used for global popular expert identification and cache initialization; the routing gating network is used for routing and activating experts; the priority management module is used for online updating of expert priorities; the prediction prefetching module is used for prediction prefetching of experts. For the specific implementation of each module, please refer to the steps corresponding to the MoE batch inference method based on priority differential caching described below. Expert parameter migration between the host and GPU is achieved via PCIe.

[0023] like Figure 2 As shown, this embodiment of the invention provides a high-throughput MoE batch inference method based on priority differential caching. The method includes the following steps.

[0024] S1: Global popular expert identification and cache initialization.

[0025] Before inference, the system analyzes the activation frequency of experts in each layer (MoE layer) of the MoE model based on statistical data from the MoE model fine-tuning phase. Experts with activation frequencies above a preset threshold in each layer are designated as globally popular experts, and a priority list for each layer is established. In the GPU, high-priority and medium-priority caches are configured independently for each layer, while all layers share a low-priority cache. The high-priority cache stores the parameters of globally popular experts for each layer; experts with activation frequencies above the threshold (globally popular experts) are permanently stored in the high-priority cache corresponding to each layer. The medium-priority cache stores the parameters of short-term popular experts in the corresponding layer, and the low-priority cache is used for dynamic loading and prefetching. All expert parameters for all layers are stored in the CPU of the host segment.

[0026] S2: Router Activation Expert.

[0027] During the inference process, the routing gating network selects the set of experts to be activated in the current layer based on the characteristics of the current batches of inputs, and outputs the expert invocation request corresponding to the set of experts to be activated in the current layer.

[0028] S3: Expert priority is updated online.

[0029] The priority management module updates the priority scores of experts based on their current activation status: if an expert (identifier) ​​is in the set of experts to be activated in the current layer, its priority score is increased; if an expert (identifier) ​​is not in the set of experts to be activated in the current layer, its priority score is decreased. The updated priority results serve as input for subsequent cache scheduling.

[0030] S4: Differential cache hierarchical scheduling.

[0031] The differential caching module migrates and replaces experts among high, medium, and low-priority caches based on priority scores and threshold ranges. Experts remain in the high-priority cache, while other experts are migrated from the low-priority cache to the medium-priority cache if their priority scores exceed a preset cache threshold; otherwise, they are released from the medium-priority cache if they have not been accessed for a long time, thus achieving dynamic allocation and hierarchical optimization of cache resources.

[0032] S5: Prefetching.

[0033] The prediction prefetching module predicts the set of experts that may need to be activated in the next layer based on the gating results of the current layer (the set of experts that need to be activated in the current layer) and the historical activation trajectory, and asynchronously loads them into a low-priority cache during the computation of the current layer. When the prediction hits, the experts required for the next layer do not need to be loaded from the host again, achieving overlap between computation and communication. The next layer is taken as the current layer, and the process returns to S2 for the next layer iteration until all rounds of the MoE model have been completed. The output of the last layer in the current round is used as the inference result of the current round, and the inference result of the current round is used as the input of the first layer of the next round for all layers of the next round iteration. When all rounds of the MoE model have been completed, the output results of the last layer iteration in each round are concatenated as the final inference result.

[0034] Specifically, in step S1, the global popular expert identification includes the following steps: S11: In the fine-tuning phase before inference begins, the system statistically analyzes the activation frequency of each expert in each layer based on historical data, establishes an activation frequency distribution table, and identifies the high-frequency experts with the highest activation frequency in each layer of the MoE model (e.g., selecting the top 2 experts in activation frequency), which are the global hot experts corresponding to each layer.

[0035] S12: Establish an independent priority list for each layer, recording the expert identifier and priority score for that layer in the priority list to guide subsequent cache scheduling. Specifically, for globally popular experts in each layer, their expert priority score is set to the preset maximum score (e.g., 2.0), and they reside permanently in the high-priority memory of that layer to avoid repeated migration during inference. For the remaining experts in each layer, their initial expert priority scores are set to the preset minimum score, and are initially stored in the host CPU. The priority lists for each layer are maintained independently but can share statistical information for global comparison. After this step, the system completes the pre-identification and cache initialization of high-frequency experts at each layer, ensuring rapid hits in the subsequent inference stage.

[0036] Step S2, after the inference phase begins, involves the routing gating network in the MoE processing unit calculating the activation probabilities of each expert required for each task at the current layer based on the features of multiple batches of input. For each task, the top-k experts with the highest activation probabilities are selected to participate in the current layer's computation (the computation between the features corresponding to each task and their corresponding expert parameters, i.e., the computation in S43). This results in a set of activated experts across multiple batches of processing (including multiple tasks being processed simultaneously). This set contains the top-k expert identifiers corresponding to the activation probabilities of each task in the multiple batches of input. Each activated expert triggers a cache hit detection request. This step primarily provides activation signals, serving as a basis for subsequent priority updates and cache scheduling.

[0037] Step S3 corresponds to the priority management module, which dynamically adjusts the priority scores of each expert in the current layer based on their activation status, achieving adaptive priority updates. The priority score update adopts a differential score mechanism, with each expert, except for globally popular experts, maintaining a real-time priority score.

[0038] Specifically, the methods for updating expert priority scores include: R3-1: The range of expert priority scores is [lowest score, highest score]; in this embodiment of the invention, the lowest score is set to 0 and the highest score is set to 2.

[0039] R3-2: The priority score of globally popular experts always remains the highest and does not require dynamic updates.

[0040] R3-3: If an expert belongs to the set of experts that need to be activated in the current layer, increase their priority score (e.g., add 1.0).

[0041] R3-4: If an expert is not activated (does not belong to the set of experts that need to be activated in the current layer) but is in the medium priority cache, reduce its priority score (for example, reduce it by 0.4).

[0042] R3-5: If an expert is not activated (does not belong to the set of experts to be activated in the current layer) and is not in the medium priority cache, reduce its priority score, but the reduction should be less than that of R3-4 (e.g., reduce by 0.2).

[0043] Through multiple iterations, the system dynamically adjusts priority scores based on score changes, forming a continuously evolving list of expert priorities. This mechanism reflects both temporal locality (the change in priority scores at each level of experts) and the differentiation of experts into popular and less popular ones through continuous updates.

[0044] Figure 3 This illustrates the storage relationship and data migration path between the host and GPU. The expert parameters of the MoE model are stored on the host, while other non-MoE parameters reside on the GPU. During runtime (inference), the expert parameters are migrated from the host to the GPU.

[0045] Figure 4 The structure of the differential cache is shown. Figure 5 The internal structure of the MoE model is shown. Each iteration of the MoE model includes iteration over all layers (e.g., n layers) in the current round. Figure 5 The diagram illustrates three iteration rounds, each containing n layers (layer 1 to layer n). The GPU-side differential cache module consists of high-priority, medium-priority, and low-priority caches. High- and medium-priority caches are configured independently within each layer, while the low-priority cache is shared across all layers. It serves as a temporary storage area for the current layer's activation experts and the next layer's prediction experts. Specifically, the low-priority cache includes a cache area for the current layer's activation experts and a cache area for the next layer's prediction experts. The cache area for the current layer's activation experts temporarily stores the expert parameters from the set of experts to be activated in the current layer. After expert migration and replacement in S44, this area is cleared to make room for loading activation experts for the next layer. For a MoE model with n layers, the differential cache module contains n high-priority caches, n medium-priority caches, and 1 low-priority cache. The i-th layer of each round corresponds to the i-th high-priority cache and the i-th medium-priority cache, where i ∈ {1, 2, ..., n}. All layers in each round share the low-priority cache. In this embodiment, the capacity of the medium-priority cache is twice that of the high-priority cache.

[0046] Step S4 specifically includes the following steps: S41: Based on the expert call request generated in S2 for the current layer, the system sequentially searches for the required expert parameters in the high, medium, and low-level caches, i.e., the parameters of the experts in the set of experts to be activated in the current layer. If a cache hit occurs in any of the high, medium, or low-level caches, the expert is ready and can be used for computation in the current layer. If the current layer is the first layer of the first round, all required expert parameters except for the globally popular expert parameters stored in the high-priority cache are on the host side; otherwise, the system determines whether the required expert parameters are in the medium or low priority cache based on the expert promotion method in S43 and the expert parameters pre-fetched from the "next layer prediction expert cache area" in the low-priority cache.

[0047] S42: If the required expert is not found in the cache, that is, if there is at least one expert parameter in the set of experts to be activated in the current layer that is not in the high, medium and low level caches, then load the expert parameter from the host to the active expert cache area of ​​the current layer in the low priority cache.

[0048] S43: Once all the experts required for this layer have been loaded, the system triggers parallel computation for this layer. The MoE processing unit on the GPU distributes the input features from multiple batches to the corresponding experts. Each expert parameter in the current layer performs inference calculations such as matrix multiplication and nonlinear activation with its corresponding input to obtain the output result of the current layer. Based on the output result of the current layer, the input features of the next layer in multiple batches are determined.

[0049] S44: After calculation, the system performs a promotion operation based on the priority scores of experts in the low-priority cache and the current layer's priority cache, as well as the cache threshold. The system sorts experts in the low-priority cache in descending order of their scores. When there is an expert in the low-priority cache with a priority score higher than the preset cache threshold, the following rules are executed: R44-1: If the medium-priority cache is not full, high-scoring experts (experts whose priority score is higher than the preset cache threshold) in the low-priority cache will be directly promoted to the medium-priority cache.

[0050] R44-2: If the medium-priority cache is full, experts in the medium-priority cache are sorted in ascending order of priority score. For experts in the medium-priority cache whose priority score is less than a preset cache threshold, the comparison starts from the lowest score: if the lowest priority score expert in the medium-priority cache is also below the cache threshold, then the highest priority score expert in the low-priority cache replaces that expert. If the priority scores of all experts in the medium-priority cache exceed the cache threshold, no replacement is performed.

[0051] R44-3: Repeat this process until there are no experts with priority scores below the threshold in the medium-priority cache, or no experts to be promoted in the low-priority cache.

[0052] S45: Clear the space of the activated experts in the low-priority cache (activated expert cache area in this layer) to make room for the next layer to load. If the expert priority score in the low-priority cache is not higher than the preset cache threshold, the activated expert cache area in this layer is cleared directly.

[0053] Through the above-mentioned layering and promotion strategy, the system realizes the retention of high-frequency experts, the dynamic retention of mid-frequency experts, and the on-demand replacement of low-frequency experts, which significantly improves cache hit rate and GPU memory utilization efficiency.

[0054] Step S5 specifically includes the following steps: S51: The system calls a GRU-based predictor for each task based on the distribution of activated experts across multiple batches in the current layer. The predictor takes the activated expert corresponding to the current task as input and combines it with the historical activation trajectory (the experts activated by the current task in layers prior to the current layer) to generate the next layer expert activation probability distribution for each task. In the current round, if the last layer of the model is reached, the system will predict the expert activation probability distribution of the first layer of the next iteration and then continue the loop.

[0055] S52: Combine the prediction results of all tasks in multiple batches (the next-level expert activation probability distribution of each task) to obtain the overall expert activation probability in the next level in multiple batch inference, and sort them from high to low probability.

[0056] S53: Based on the batch size, select the top N (Top-N) predictive experts with the highest overall activation probability in multi-batch inference for the next layer as the set of experts to be activated in the next layer, and preload them into the low-priority cache area of ​​the next layer predictive expert cache (e.g., Figure 4 (As shown on the right) (Clear the pre-activated experts of the current layer predicted by the previous layer). When the next layer of inference arrives, if the prediction is correct, the expert can directly participate in the computation without being loaded from the host, thus achieving parallelization of computation and communication. Using the next layer as the current layer, repeat S2-S5 until all layers of the MoE model have been executed. Use the output of the last layer in the current round as the inference result for the current round, and then execute the next iteration until all rounds have been completed. Concatenate the output results of the last layer iteration in each round as the final inference result.

[0057] To facilitate understanding of the technical solution of the present invention, the specific embodiments of the present invention will be further described below in conjunction with the accompanying drawings and specific examples. It should be understood that the following embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0058] See Figure 6This embodiment provides a specific application scenario of MoE batch inference based on priority differential caching, demonstrating the parameter setting, cache layering and expert scheduling process of the system in the multi-round inference process.

[0059] In this embodiment, the system sets the priority parameters as follows: The highest score is 2.0, the cache threshold is 1.0, activation adds 1.0 points, inactive cache deducts 0.4 points, and inactive cache deducts 0.2 points.

[0060] The GPU side includes high-priority cache, medium-priority cache, and low-priority cache. The high-priority cache is used to store globally popular experts, the medium-priority cache is used to store locally popular experts, and the low-priority cache is divided into an expert activation cache area for the current layer and a prediction expert cache area for the next layer, which are used for loading active experts in the current layer and prefetching prediction experts in the next layer, respectively.

[0061] In this embodiment, the top-ranked expert in each layer is selected as the globally popular expert, meaning the high-priority cache capacity is equal to that of one expert. The medium-priority cache capacity is set to twice that of the high-priority cache, meaning the medium-priority cache capacity is equal to that of two experts. The low-priority cache capacity depends on the number of experts in a MoE layer. It is necessary to ensure that, in extreme cases where the number of tasks in a batch is huge and all experts are activated, the cache area of ​​activated experts in this layer (high-priority + medium-priority + low-priority caches) can hold all the experts in a layer. However, the space used in actual use depends on the set of activated experts.

[0062] At the start of inference, the host computer initializes priorities using statistics from the fine-tuning phase, resident high-priority experts in the high-priority cache. In the illustrated MoE layer, the globally popular expert number is 0.

[0063] This embodiment demonstrates a two-round reasoning process.

[0064] In the first round of reasoning, the activated experts output by the gating network in this round are expert 0, expert 4, expert 5 and expert 6.

[0065] Among them, expert 0 hit the high-priority cache, expert 4 hit the next layer of predicted expert cache area of ​​the low-priority cache, and experts 5 and 6 missed the cache.

[0066] For experts that are not hit, the host loads them into the GPU's low-priority cache, which is the same level of expert activation cache, via the PCIe channel.

[0067] Once all activation experts have been loaded, the MoE processing unit initiates batch computation. All activation experts perform their corresponding matrix calculations and nonlinear operations in parallel to obtain the output results of the current layer.

[0068] The priority management module updates scores based on cache hit status and expert activation status. Hit experts maintain or increase scores, newly loaded experts increase scores by 1.0, and inactive experts decrease scores.

[0069] In this round, Expert 0 is the most popular expert globally, and its score remains at the highest level of 2.0 without being updated. Experts 4, 5, and 6 receive bonus points. The priority scores of all other experts are reduced.

[0070] After the calculation is complete, experts whose scores in the low-priority cache exceed the cache threshold of 1.0 will undergo a promotion process: If the medium-priority cache is not full, it will be promoted directly; if the medium-priority cache is full, the experts will be sorted in ascending order of their scores, and the comparison will start from the expert with the lowest score. If the expert's score is lower than the threshold, it will be replaced by the expert with the highest score in the low-priority cache.

[0071] After this update, Expert 1's score in the medium-priority cache dropped to 0.8, below the cache threshold, and was therefore replaced by Expert 4, which had the highest priority score in the low-priority cache. Expert 3's score dropped to 1.2, but remained above the cache threshold, so it stayed in the medium-priority cache.

[0072] After the promotion process, Expert 4 was moved to the medium-priority cache, while Expert 1 was evicted from the medium-priority cache. Experts 5 and 6, who failed to be promoted, were cleared from the low-priority cache.

[0073] Simultaneously, the prediction prefetch module, based on the current layer's activation distribution and historical activation trajectories, invokes a GRU-based prediction model to predict the set of experts that may be activated in the next layer. The prediction result is expert 3′ for the next layer. Expert 3′ is asynchronously loaded by the host into the low-priority cache of the next-layer prediction expert buffer during the current layer's computation phase.

[0074] Before the second round of inference, the previous layer predicted that expert 2 would be hit in this round and preloaded it into the next layer of the low-priority cache area for predicting experts.

[0075] In the second round of inference, the gating network outputs activation experts as expert 0, expert 2, expert 3, and expert 7. The system then checks the cache hit status sequentially.

[0076] Expert 0 hit the high-priority cache, expert 3 hit the medium-priority cache, expert 2 hit the low-priority cache, and expert 7 missed the cache.

[0077] The expert 7 that was not hit is loaded into the low-priority cache of the same layer of the expert activation cache by the host.

[0078] Once all activated experts have been loaded, the system executes parallel computation for the current batch. After the computation is complete, the priority management module updates the expert scores according to the rules. The scores for Expert 2, Expert 3, and Expert 7 are increased.

[0079] Among them, Expert 3's score was 1.2 before the bonus points were added, and reached 2.2 after the bonus points were added, which exceeded the maximum score. Therefore, the maximum score was limited to 2.0.

[0080] After the score adjustment, the scores of Expert 4 and Expert 3 in the medium-priority cache are both higher than the cache threshold, so this round will not trigger the replacement of the medium-priority cache by the low-priority cache.

[0081] During the second round of calculation, the prediction prefetch module performs the next layer prediction again based on the new activation information. The next layer prediction will use expert 5′ and load it asynchronously into the next layer prediction expert cache area with low priority cache, in preparation for the next layer inference call.

[0082] Through the above implementation process, the system sequentially completes activation detection, expert loading, computation execution, priority update and prediction prefetching in each round of inference, achieving time overlap between loading and computation.

[0083] Among them, the high-priority cache resides in the global high-frequency experts for a long time, the medium-priority cache is used to store short-term hot experts, and the low-priority cache is used for dynamic loading and prefetching.

[0084] The priority management module differentiates experts based on the increase or decrease of scores, while the differential caching module achieves efficient reuse of cache space through hierarchical migration.

[0085] This architecture effectively reduces the number of parameter migrations from the host to the GPU, and improves cache hit rate and memory utilization efficiency in multi-batch scenarios, thereby improving the overall throughput performance of batch MoE inference.

[0086] Example 2 This invention provides a MoE batch inference processor based on priority differential caching, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the MoE batch inference method based on priority differential caching in Embodiment 1 above.

[0087] The relevant technical solutions are the same as above, and will not be repeated here.

[0088] Example 3 This invention provides a computer program product, including a computer program that, when run on a computer, causes the computer to execute the steps of the MoE batch inference method based on priority differential caching in Embodiment 1 above.

[0089] The relevant technical solutions are the same as above, and will not be repeated here.

[0090] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A MoE batch inference method based on priority differential caching, characterized in that, include: S1: Based on the activation frequency of experts in each layer of the MoE model, identify the global hot experts for the corresponding layer, and establish an independent priority list for experts in each layer. The priority list records the priority score of the experts in that layer. The priority score of the global hot experts is the highest score, and the initial priority score of the other experts in that layer is the lowest score. In the GPU, high and medium priority caches are configured independently for each layer, and a low priority cache is configured for all layers. The high priority cache is used to store the parameters of the global hot experts. S2: In the current round of the inference phase, the routing gating network in the MoE model selects the set of experts to be activated in the current layer based on the characteristics of the multiple batches of inference inputs in the current layer. S3: If the expert of the current layer is in the expert set, then increase the priority score of the expert; otherwise, decrease the priority score of the expert to obtain the updated priority score of the expert of the current layer. S4: Determine whether the expert parameter in the expert set can be found in any of the high, medium, and low priority caches. If yes, the expert is loaded. If not, the expert parameter is loaded from the CPU into the active expert cache area of ​​the current layer in the low priority cache, thus completing the expert loading. After all experts in the expert set are loaded, a parallel inference calculation of the features of the current layer's multi-batch inference input and all expert parameters of the current layer is triggered to obtain the current layer's output result, thereby obtaining the features of the next layer's multi-batch inference input. When the expert priority score in the active expert cache area of ​​the current layer is higher than the preset cache threshold, if the medium priority cache is not full, the expert parameters with high priority scores in the low priority cache are migrated to the medium priority cache. If the medium priority cache is full, the expert parameters with high priority scores in the low priority cache are used to replace the expert parameters in the medium priority cache with expert priority scores lower than the preset cache threshold according to their priority scores from low to high. Then, the active expert cache area of ​​the current layer is cleared. S5: Predict the set of experts to be activated in the next layer and load it into the next layer prediction expert cache in the low-priority cache; take the next layer as the current layer and return to S2 for iteration until all layers of the MoE model have been iterated, and take the output of the last layer as the inference result of the current round.

2. The MoE batch inference method based on priority differential caching according to claim 1, characterized in that, In S3, reducing the expert's priority score includes: If the expert parameter is located in the medium-priority cache, reduce the expert's priority score by one point; If the expert parameter is not in the medium priority cache, reduce the expert's priority score by a second score, where the second score is less than the first score.

3. The MoE batch inference method based on priority differential caching according to claim 1 or 2, characterized in that, S2 includes: The routing gating network in the MoE model calculates the activation probability of each expert required for each task in the current layer based on the characteristics of multiple batches of input in the current layer. The set of the k experts with the highest activation probabilities for all tasks in the current layer's multiple batches of inference is taken as the expert set.

4. The MoE batch inference method based on priority differential caching according to claim 3, characterized in that, In S1, the global popular experts of the corresponding layer are identified based on the expert activation frequency of each layer of the MoE model. This includes: in the fine-tuning stage before inference, the expert activation frequency in each layer of the MoE model is counted, and experts whose expert activation frequency is higher than a preset threshold are identified as the global popular experts of the corresponding layer.

5. The MoE batch inference method based on priority differential caching according to claim 1 or 4, characterized in that, Predict the set of experts that need to be activated in the next layer, including: The experts to be activated for each task in the current layer of the expert set to be activated, as well as the historical expert activation distribution data for that task, are input into the GRU-based predictor to obtain the next layer expert activation probability distribution for that task. The expert activation probability distributions of each task in the current layer multi-batch inference are merged to obtain the set of expert activation probability distributions in the next layer multi-batch inference. Based on the batch size, select the top N experts with the highest expert activation probability distribution from the expert activation probability distribution set as the next layer of experts to be activated in the prediction.

6. The MoE batch inference method based on priority differential caching according to claim 1, characterized in that, The capacity of the medium-priority cache is twice that of the high-priority cache.

7. A MoE batch inference system based on priority differential caching, characterized in that, The GPU includes: a preprocessing module, a priority management module, a differential cache module, and a prediction prefetch module; The preprocessing module is used to identify the global hot experts for each layer based on the activation frequency of experts in each layer of the MoE model, and to establish an independent priority list for experts in each layer. The priority list records the priority score of the experts in that layer, with the priority score of the global hot experts being the highest score, and the initial priority scores of the other experts in that layer being the lowest scores. It is also used to configure high and medium priority caches independently for each layer, and to configure a shared low priority cache for all layers. The high priority cache is used to store the parameters of the global hot experts. In the current round of the inference phase, the routing gating network in the MoE model selects the set of experts to be activated in the current layer based on the characteristics of the multiple batches of inference inputs in the current layer. The priority management module is used to update the expert priority score of the current layer, including: if the expert of the current layer is in the expert set, then increase the priority score of the expert; otherwise, decrease the priority score of the expert, so as to obtain the updated expert priority score of the current layer. The differential caching module is used to determine whether the expert parameter in the expert set can be found in any of the high, medium, and low priority caches. If so, the expert is loaded. If not, the expert parameter is loaded from the CPU into the active expert cache area of ​​the current layer in the low priority cache, thus completing the expert loading. After all experts in the expert set are loaded, a parallel inference calculation of the features of the current layer's multi-batch inference input and all expert parameters of the current layer is triggered to obtain the current layer's output result, thereby obtaining the features of the next layer's multi-batch inference input. It is also used to migrate expert parameters with high priority scores in the low priority cache to the medium priority cache if the medium priority cache is not full, and if the medium priority cache is full, replace expert parameters with high priority scores in the medium priority cache with expert parameters with high priority scores in the low priority cache according to the priority scores from low to high. After that, the active expert cache area of ​​the current layer is cleared. The prediction prefetching module is used to predict the set of experts to be activated in the next layer and load it into the next layer prediction expert cache area in the low priority cache. The next layer is used as the current layer, and the routing gating network, priority management module, differential cache module and prediction prefetching module are repeatedly executed until all layers of the MoE model are iterated. The output of the last layer is used as the inference result of the current round.

8. A MoE batch inference processor based on priority differential caching, characterized in that, Includes computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the MoE batch inference method based on priority differential caching as described in any one of claims 1-6.

9. A computer program product, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the MoE batch inference method based on priority differential caching as described in any one of claims 1-6.

10. A MoE batch inference collaborative processing system based on priority differential caching, characterized in that, include: CPU and GPU; The CPU is used to store expert parameters for each layer of the MoE model; The GPU is used to execute the MoE batch inference method based on priority differential caching as described in any one of claims 1-6.

Images