A GPU-CPU heterogeneous collaborative inference method and system for a high-efficiency trusted recommendation-oriented video memory-restricted hybrid expert model

Abstract

The application discloses a GPU-CPU heterogeneous collaborative reasoning method and system for a mixed expert model with limited display memory for efficient and reliable recommendation, and belongs to the technical field of artificial intelligence model reasoning, reliable recommendation service and GPU-CPU heterogeneous computing resource scheduling. The method comprises the following steps: obtaining a MoE reasoning request, and constructing a reliable routing signature of the request; performing routing-aware request clustering, and aggregating into routing homogeneous batches; reading batch-level reliable hot expert priors, and performing expert residence and prefetch scheduling; adopting a sub-block-level partial replacement mechanism for expert parameters, and only migrating required expert parameter sub-blocks; constructing an expert-level task, executing GPU resident experts and high-demand experts; processing cold experts and long-tail experts through a GPU-CPU collaborative execution mechanism, completing MoE layer reasoning calculation, and outputting a recommendation result. The method performs reliable reasoning optimization around MoE routing behavior and GPU-CPU resource state at the model service layer.

Description

Technical Field

[0001] This invention belongs to the technical fields of artificial intelligence model reasoning, trusted recommendation services, and GPU-CPU heterogeneous computing resource scheduling. Specifically, it relates to a GPU-CPU heterogeneous collaborative reasoning method and system for memory-constrained hybrid expert models oriented towards efficient and trusted recommendation. Background Technology

[0002] With the application of large language models in recommender systems, the task of reliable recommendation is expanding from traditional candidate retrieval and ranking to scenarios such as generative recommendation, recommendation reason generation, candidate service re-ranking, and interactive decision support. These tasks typically involve a large model receiving user requests, candidate objects, historical interaction summaries, or sequence context, and outputting recommendation results or explanations. Therefore, it not only requires inference throughput and response latency to meet the needs of online services, but also demands that the stability, relevance, and quality control of the recommendation results be maintained as much as possible during the accelerated inference process. Mixture-of-Experts (MoE) models, by activating a small number of expert subnetworks for each input token, reduce the computational cost per iteration while maintaining a large model capacity, making them suitable for handling complex reliable recommendation inference tasks. However, their routers dynamically select experts for different requests, tokens, and layers, resulting in sparsity, non-uniformity, and time-varying characteristics in the inference process.

[0003] It should be noted that the text semantics, user interaction information, and behavior sequence information in the trusted recommendation task are inputs from the recommendation business side, and can be encoded and fused by existing recommendation models, large language model prompt templates, or upstream feature engineering.

[0004] However, while hybrid expert models employ sparse activation mechanisms to reduce computation, they still require storing all expert parameters, resulting in a total parameter size that is typically much larger than that of dense models with equivalent computational requirements. In local device, edge device, and consumer-grade GPU deployment environments for trusted recommendations, GPU memory capacity is insufficient to fully accommodate all expert parameters. Simply residing all expert parameters on the GPU would render the model undeployable, while frequent loading of experts from host memory to GPU memory would make PCIe (Peripheral Component Interconnect Express) transfers and memory replacements the main bottlenecks for recommendation inference services.

[0005] To alleviate GPU memory pressure, existing methods typically employ mechanisms such as expert offloading, expert prefetching, heat-aware caching, or CPU-side cold expert execution. While these methods can reduce memory usage to some extent, they often use historical routing trajectories or expert heat as isolated empirical indicators, making it difficult to simultaneously consider early routing signals of the current recommendation request, global high-frequency expert interference, expert reuse relationships within a batch, router score strength, and CPU / GPU resource status. Therefore, in trusted recommendation tasks, problems such as inaccurate prefetching, cold expert loading blocking, and low-contribution expert computation slowing down the critical path can easily occur, affecting the throughput, response latency, and quality stability of recommendation inference.

[0006] Specifically, existing expert offloading methods for MoE recommendation inference have the following shortcomings:

[0007] First, the expert routing paths for different recommendation requests differ significantly. When batches are formed based on length or arrival order, the expert set within a batch expands, leading to an increase in the frequency of cold expert exchanges.

[0008] Secondly, the original historical popularity is easily dominated by high-frequency experts, which cannot accurately characterize low-frequency experts but have discriminative significance for a certain type of recommendation request, thus reducing the credibility of expert predictions.

[0009] Third, prefetching and evict are often based solely on expert-level popularity, without considering the current memory resident mask, parameter sub-block requirements, and transmission costs, which can easily lead to redundant data movement.

[0010] Fourth, if GPU-CPU collaborative execution lacks routing scores and execution latency constraints, low-contribution cold expert computation may block the GPU critical path and cause unnecessary impact on the stability of recommendation results.

[0011] Therefore, how to transform observable early routing results, historical expert coverage, router scores, expert residency status, and GPU-CPU execution costs during the MoE inference process into computable, verifiable, and schedulable trusted features without changing the main structure, business feature modeling method, and recommendation semantics of the trusted recommendation model, and how to use these trusted features to guide the formation of homogeneous routing batches, hot expert residency, expert parameter sub-block migration, and GPU-CPU collaborative execution in memory-constrained environments, reduce intra-batch cold expert exchanges and redundant parameter transmissions, improve the inference throughput of the MoE trusted recommendation model, and maintain the basic stability of recommendation quality has become an urgent technical problem to be solved in current local and edge large-model recommendation inference services. Summary of the Invention

[0012] To address the problems of existing technologies, this invention proposes a GPU-CPU heterogeneous collaborative inference method and system for memory-constrained hybrid expert models aimed at efficient and reliable recommendation. This method encapsulates reliable recommendation as an online inference task carried by a large MoE model: the text, interaction information, and sequence information in the recommendation system are encoded and processed by the upstream recommendation model or the large model. This invention does not model or fuse these business-side text, interaction, and sequence information themselves, but instead optimizes reliable inference at the model service layer around MoE routing behavior and GPU-CPU resource status.

[0013] Technical Solution: To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0014] A GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models aimed at efficient and reliable recommendation includes the following steps:

[0015] Step 1: Obtain the MoE inference request for the trusted recommendation task, and construct the trusted route signature of the request based on the early layer routing results, historical expert coverage information and expert routing scores.

[0016] Step 2: Cluster route-aware requests based on the route affinity represented by the trusted route signature, and aggregate recommendation requests with similar expert access patterns and reusable value into route homogeneous batches.

[0017] Step 3: Based on the cluster identity of the homogeneous batch of routes, read the batch-level trusted thermal expert prior from the offline clustering-hierarchical thermal expert statistics table, and combine it with the current GPU memory resident status and GPU-CPU runtime resource status to determine the memory budget of fixed thermal experts, expert prefetching and non-fixed experts, so as to provide a scheduling basis for subsequent sub-block level partial replacement.

[0018] Step 4: Use a sub-block level partial replacement mechanism for expert parameters, and migrate only the expert parameter sub-blocks required for the current inference critical path between GPU memory and host memory;

[0019] Step 5: Perform online correction of batch-level trusted scheduling characteristics based on actual routing results, construct expert-level tasks, and adopt a ready-first out-of-order expert execution strategy to prioritize the execution of GPU-resident experts and high-demand experts.

[0020] Step 6: Process cold experts and long-tail experts through GPU-CPU collaborative execution mechanism, and complete the MoE layer inference calculation under the recommendation quality constraint by combining score threshold skipping and CPU early stopping strategy, and output recommendation results.

[0021] Preferably, in step 1, the specific implementation details are as follows:

[0022] Step 101: Execute a Layer 0 lightweight router probe on the incoming MoE inference request to obtain the top-k expert set corresponding to each token;

[0023] Step 102: Construct the expert frequency vector: Summarize the routing results of MoE inference requests at layer 0 into an expert frequency vector;

[0024] Step 103: Calculate the expert's request coverage number and smooth inverse coverage weight, and perform Expert-IDF reweighting and normalization on the expert frequency vector to obtain the trusted route signature.

[0025] Preferably, in step 2, the specific implementation details are as follows:

[0026] Route-aware request clustering uses trusted route signatures as the clustering basis to transform the MoE routing behavior of recommendation requests into batch-level expert reusable features, specifically:

[0027] During the offline phase, the route signatures of historical requests are clustered into For each cluster, obtain the corresponding cluster centroid. ;

[0028] During the online phase, for the arriving requests Based on its trusted route signature The cosine similarity with each cluster centroid is assigned to the nearest cluster centroid, specifically:

[0029] ;

[0030] in, This indicates the nearest cluster number to which request r is assigned; Indicates the candidate cluster number; This represents the trusted route signature obtained by reweighting and normalizing request r using Expert-IDF; Let represent the centroid of the k-th cluster;

[0031] Then, the request Add to the corresponding bucket queue In the middle; when the bucket queue reaches the target batch size or the request waiting time reaches the preset waiting time boundary, the request is taken out from the bucket queue to form a homogeneous batch of routes.

[0032] Preferably, in step 3, the specific implementation details are as follows:

[0033] Step 301: For each cluster and each MoE layer, calculate the expert trust score based on the historical routing logs, and record the Top-M hot expert set corresponding to the cluster and the layer.

[0034] Step 302: Obtain the layer-by-layer hot expert set from the offline routing statistics. The expert scheduler marks the experts in the layer-by-layer hot expert set as fixed experts.

[0035] Step 303: The expert scheduler combines batch-level trusted thermal expert priors, the current GPU memory resident status, and the GPU-CPU resource status to allocate GPU memory budget, and determines the thermal expert fixing, expert prefetching, and expert replacement strategies.

[0036] Preferably, in step 4, the specific implementation details are as follows:

[0037] Step 401: For each expert module, manage the expert parameters according to the sub-block granularity. Divide the expert parameters into three memory-dominant sub-blocks: gate_proj, up_proj, and down_proj, and record the resident status of the sub-blocks using a mask.

[0038] Step 402: Implement sub-block-level partial replacement using the Batch-Pinned Partial LRU mechanism.

[0039] Preferably, in step 5, the specific implementation details are as follows:

[0040] In each layer of inference, an expert-level task group is constructed based on the actual routing results of the router. Indicates entering the first A collection of experts that already resides in the GPU memory at the layer level;

[0041] First, execute all procedures with experts. The relevant tasks enable the GPU to start computing immediately, reducing idle time caused by waiting for cold experts to load;

[0042] For the remaining non-resident experts, statistics are as follows: Layered experts The number of tokens allocated is denoted as and in accordance with Construct expert queues from largest to smallest. ;

[0043] Expert scheduler prioritizes The top few experts with a high hit count initiate GPU prefetching, allowing high-demand experts to be loaded into GPU memory as early as possible.

[0044] Preferably, in step 6, the specific implementation details are as follows:

[0045] For experts who are not currently residing in the queue but have high demand, the GPU will retrieve them from the expert queue. The process begins with the head of the queue, loading its parameter block into GPU memory before execution; for long-tailed cold experts that are not resident and have a small hit count, the CPU loads them from... Processing begins at the tail of the queue, and forward propagation is completed directly using the CPU-side resident weights.

[0046] The GPU and CPU execute their respective assigned expert tasks in parallel and synchronize at the output merging point.

[0047] As a preferred option, the score threshold skipping strategy is as follows:

[0048] For experts assigned to CPU processing Use its aggregated router scores Measure the expert's contribution to the final output; when At that time, skip the expert's calculations, where, The score threshold determined offline can be configured according to the accuracy target.

[0049] As a preferred option, the CPU early termination strategy is as follows:

[0050] When the remaining tasks are better suited to be handled by the GPU, the CPU stops consuming tasks and hands over the remaining tasks to the GPU.

[0051] Specifically, experts were estimated separately. The processing latency on the GPU and CPU is calculated using the following formula:

[0052] ;

[0053] ;

[0054] in, Indicates that experts The estimated time for loading parameters into GPU memory. and They represent experts respectively. Processing the forward time of a single token on both GPU and CPU This indicates the additional overhead of CPU execution; This represents the number of tokens retained and allocated to the l-th level expert e after the score threshold filtering. This represents the estimated total latency for expert e to complete parameter loading and forward computation on the GPU; This represents the estimated total latency after expert e completes the forward computation on the CPU and accounts for additional overhead.

[0055] A GPU-CPU heterogeneous collaborative inference system for memory-constrained hybrid expert models aimed at efficient and reliable recommendation is provided. The system implements the GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models aimed at efficient and reliable recommendation as described in any of the preceding claims. The system includes: a reliable scheduling feature construction module, a route-aware request batch processing module, an expert scheduling module, and a GPU-CPU collaborative inference module.

[0056] Trusted scheduling feature construction module: responsible for transforming early layer routing results, historical coverage, routing scores, batch-level trusted hot expert priors, expert residency status, and execution costs into unified inference scheduling features;

[0057] Route-aware request batch processing module: Generates route homogeneous batches based on the early-layer route affinity represented by trusted route signatures;

[0058] Expert scheduling module: performs expert fixing, pre-fetching, and sub-block-level replacement based on batch-level trusted thermal expert priors;

[0059] GPU-CPU collaborative reasoning module: Performs expert-level task scheduling based on the actual routing results, and maps resident experts, high-demand experts, and long-tail cold experts to appropriate computing resources respectively.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0061] (1) The present invention integrates early layer routing affinity, historical request coverage information, expert routing score and expert activation statistics into trusted scheduling features, and forms a homogeneous batch of routes for recommendation requests accordingly, so that the batch processing basis changes from request length or arrival order to expert reuse relationship, reduces the number of different experts in the batch, improves the expert reuse rate, and reduces the frequency of cold expert exchange.

[0062] (2) The present invention combines batch-level trusted thermal expert priors, GPU memory resident state and expert parameter sub-block state, and migrates only the main parameter sub-blocks required for inference, avoiding redundant data movement caused by complete expert replacement and reducing DRAM-VRAM transmission overhead.

[0063] (3) This invention uses hot expert fixation, Batch-Pinned Partial LRU and computation critical path priority prefetching mechanism to enable offline expert access statistics, online batch features and memory resource status to work together on expert residency decision, improve the effectiveness of expert residency in GPU memory and reduce the blocking of inference critical path by expert loading.

[0064] (4) This invention combines GPU-resident expert execution, CPU long-tail cold expert processing, out-of-order execution, score threshold skipping and CPU early stopping strategy, so that the routing score credibility, expert hit scale and CPU / GPU execution cost jointly determine the cold expert processing path, thereby improving the throughput of MoE trusted recommendation inference under memory-constrained environment.

[0065] (5) While maintaining the basic stability of the recommendation output quality, the present invention can effectively improve the service efficiency of MoE recommendation model on local and edge platforms. It is suitable for large language model service scenarios where GPU memory is limited but CPU and host memory resources can be used in a coordinated manner and efficient and reliable recommendation reasoning is required. Attached Figure Description

[0066] Figure 1 This is a system overview diagram of the GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models, which is aimed at efficient and reliable recommendation in this invention.

[0067] Figure 2 This is a comparison chart of the activation popularity of each batch of experts before and after clustering.

[0068] Figure 3 This is a layer-by-layer similarity comparison chart of different request activation expert sets on GPT-OSS and DeepSeek-MoE;

[0069] Figure 4 This is a schematic diagram of the route-aware request batch processing process of the present invention;

[0070] Figure 5 This is a schematic diagram of the decomposition of the expert parameter submodule of the MoE model;

[0071] Figure 6 This is a comparison chart of the execution process of sub-block level partial replacement and whole expert replacement;

[0072] Figure 7 This is a schematic diagram illustrating the process of GPU and CPU executing cold expert and hot expert tasks in parallel. Detailed Implementation

[0073] The present invention will be further illustrated below with reference to specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0074] Example 1

[0075] This embodiment provides a GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models aimed at efficient and reliable recommendation. It optimizes the inference service of a large MoE model carrying a reliable recommendation task. Without altering the recommendation model structure, business feature fusion method, routing results, or expert output aggregation semantics, it transforms early layer routing results, historical expert coverage information, router scores, and runtime resource states into reliable scheduling features. Based on these reliable scheduling features, it completes the memory-constrained MoE recommendation inference service. This invention focuses on optimizing the inference service layer of the large recommendation model during runtime, specifically how to maintain high throughput, low transmission overhead, and quality stability for recommendation inference under memory-constrained conditions.

[0076] The trustworthy recommendation task is carried out by a Mixture-of-Experts (MoE) large model, which is used to generate recommendation results, re-ranking results or recommendation reasons based on user requests, candidate objects and recommendation context. This method does not change the recommendation model structure, business feature fusion method and recommendation semantics, but optimizes the inference service efficiency and output quality stability of this task in a memory-constrained environment.

[0077] like Figure 1 As shown, the method specifically includes the following steps:

[0078] Step 1: Obtain the MoE inference request for the trusted recommendation task, and construct the trusted route signature of the request based on the early layer routing results, historical expert coverage information and expert routing scores.

[0079] The trusted route signature is not the text, interaction, or sequence feature itself in the trusted recommendation service, but rather a service scheduling feature formed by the early layer routing results of the current request, the sparsity of expert coverage, and the router score during the MoE recommendation inference process.

[0080] In practice, a Layer 0 lightweight router probe is first executed on the arriving trusted recommendation requests to obtain the top-k expert set corresponding to each token. This process only performs route calculations and does not execute expert FFN, thus obtaining the early routing behavior of the current request with minimal overhead. Then, an expert frequency vector is constructed, and the historical coverage distribution is reweighted using Expert-IDF to suppress the interference of common experts on similarity, resulting in a normalized trusted route signature.

[0081] Specifically as follows:

[0082] Step 101: First, execute the Layer 0 lightweight router probe on the incoming MoE inference request for the trusted recommendation task to obtain the top-k expert set corresponding to each token;

[0083] For the input MoE inference request Suppose the request contains Each token runs a lightweight router probe at the 0th MoE layer; for each token... Router probe outputs top-k expert set And it does not implement the expert feedforward network FFN, where, Indicates a collection of experts. ;

[0084] Step 102: Construct the expert frequency vector: This involves processing the MoE inference request. The routing results at layer 0 are aggregated into an expert frequency vector. , of which The frequency of each expert is expressed as:

[0085]

[0086] in, Indicates MoE inference request Selecting an expert during Layer 0 routing Number of times, This indicates an indicator function that takes the value 1 when the condition is true and 0 otherwise. This indicates the number of tokens contained in request r; This represents the set of top-k experts selected by the lightweight router probe at the 0th MoE layer for the t-th token in request r.

[0087] Step 103: Calculate the expert's request coverage number and smooth inverse coverage weight, and perform Expert-IDF reweighting and normalization on the expert frequency vector to obtain the trusted route signature.

[0088] To reduce the dominant role of global high-frequency experts in the calculation of recommendation request similarity, an Expert-IDF reweighting mechanism is adopted to optimize the early routing features for trusted scheduling. Specifically, the routing results of the 0th layer are summarized into an expert frequency vector, which is then reweighted and normalized using Expert-IDF to obtain the request-level trusted route signature.

[0089] For including An offline window for historical requests, computing expert Request coverage and smoothing inverse coverage weights Specifically:

[0090]

[0091]

[0092] in, Indicates coverage experts in historical requests The number of requests, This represents the smoothed inverse coverage weight of expert e, used to quantify the scarcity of that expert in historical requests; Follow Increase and decrease, used to reduce the weight of general experts and increase the influence of low-frequency experts in route similarity calculation; Indicates an indicator function; Indicates MoE inference request Selecting an expert during Layer 0 routing The number of times.

[0093] The expert frequency vector is reweighted and normalized using Expert-IDF, thereby forming a trusted route signature that takes into account both the current recommendation request route and the historical expert coverage distribution, which is the trusted route signature of the request, specifically:

[0094]

[0095]

[0096] in, This represents the e-th expert frequency component after Expert-IDF reweighting; Indicates MoE inference request Selecting an expert during Layer 0 routing The number of times; This represents the smoothed inverse cover weight of expert e; This represents the normalized trusted route signature for request r; This represents a vector composed of the frequency components of each weighted expert.

[0097] In this embodiment, a request is made. and requests The similarity of the Layer 0 routes between them is calculated using cosine similarity, specifically:

[0098]

[0099] in, This represents the route similarity between request i and request j at layer 0; This indicates the trusted route signature for request i; This represents the trusted route signature for request j.

[0100] In general In the layer, token-level routing decisions are aggregated into a normalized expert counting vector. ,ask With request The expert activation similarity between them is represented as:

[0101]

[0102] in, This represents the similarity of expert activations between request i and request j at layer l. This represents the normalized expert count vector of request i at layer l; Let represent the normalized expert count vector of request j at layer l.

[0103] Step 2: Cluster route-aware requests based on the route affinity represented by the trusted route signature, and aggregate recommendation requests with similar expert access patterns and reusable value into route homogeneous batches to reduce the types of experts activated within a batch.

[0104] Furthermore, in the online phase, the trusted route signature of the recommended request is matched with the offline clustering center using cosine similarity, and then the request is assigned to the corresponding clustering bucket. The system forms a batch when the number of requests in a bucket reaches the target batch size or the waiting time reaches a preset limit. Since requests in the same bucket are more similar in terms of expert activation mode and low-frequency expert identification, the formed batches can reduce the number of experts activated within the batch and improve the expert reuse rate.

[0105] Route-aware request clustering uses trusted route signatures as the clustering basis to transform the MoE routing behavior of recommendation requests into batch-level expert reusable features, as follows:

[0106] During the offline phase, the route signatures of historical requests are clustered into For each cluster, obtain the corresponding cluster centroid. .

[0107] During the online phase, for the arriving requests Based on its trusted route signature The cosine similarity with each cluster centroid is assigned to the nearest cluster centroid, specifically:

[0108]

[0109] in, This indicates the nearest cluster number to which request r is assigned; This represents the candidate cluster number, with a value ranging from 1 to K; This represents the trusted route signature obtained by reweighting and normalizing request r using Expert-IDF; This represents the centroid of the k-th cluster.

[0110] Then, the request Add to the corresponding bucket queue In the middle; when the bucket queue reaches the target batch size or the request waiting time reaches the preset waiting time boundary, the request is taken out from the bucket queue to form a homogeneous batch of routes.

[0111] In this way, instead of mechanically batching requests based on request length or arrival time, the present invention integrates early-layer routing, historical coverage, and clustering priors into batch-level trusted routing features. This ensures that requests within the same batch maintain similar expert activation modes and expert reuse requirements. This feature directly reduces the number of different activated experts within each batch, improves expert reuse rate within a batch, and reduces the frequency of cold experts being exchanged between host memory and GPU memory under memory constraints, thereby providing a stable basis for subsequent expert residency, prefetching, and replacement decisions.

[0112] In this invention, the trusted route signature is a request-level feature obtained by reweighting and normalizing the Level 0 expert frequency vector of a single request using Expert-IDF; the batch-level trusted route feature is a batch-level route representation formed by the trusted route signatures of requests within the same cluster bucket and their cluster identities; the trusted scheduling feature is a higher-level concept, which also includes batch-level trusted hot expert priors, expert residency states, and GPU-CPU execution costs.

[0113] Step 3: Based on the cluster identity of the batch, read the batch-level trusted thermal expert prior from the offline clustering-hierarchical thermal expert statistics table, and combine it with the current GPU memory resident status and GPU-CPU runtime resource status to determine the memory budget of fixed thermal experts, expert prefetching and non-fixed experts, so as to provide a scheduling basis for subsequent sub-block level partial replacement.

[0114] This step is used to inherit the homogeneous batch of routes formed in step 2 and establish a fixed expert set, a prefetch queue, an evictionable expert pool, and a memory budget for the sub-block-level partial replacement in step 4. Specifically, the system reads the Top-M hot expert set layer by layer according to the batch cluster identity, prioritizes fixing or prefetching highly reusable experts, and then step 4 performs sub-block-level partial replacement only on non-fixed experts. Therefore, steps 2 to 4 form a continuous link of "homogeneous batch formation of routes - prior reading and resident scheduling of batch-level hot experts - sub-block-level partial replacement"; this step includes the identification of the hottest expert in the batch, fixing experts, dynamic expert prefetching, and in-memory expert scheduling.

[0115] Once a batch is formed, the system reads the batch-level trusted hot expert prior (i.e., a layer-by-layer Top-M hot expert set organized by cluster identity and MoE layer, obtained from offline routing log statistics) based on the batch cluster identity. The expert scheduler marks the experts in this prior as fixed experts. The main parameter blocks of fixed experts remain resident in GPU memory and do not participate in subsequent evictions, thus enabling offline routing statistics and the current batch cluster identity to jointly serve the rapid execution of high-frequency experts.

[0116] Specifically as follows:

[0117] Step 301, Offline Phase: For each cluster and each MoE layer, calculate the expert trust score based on historical routing logs, and record the Top-M hot expert set corresponding to the cluster and the layer.

[0118] The expert credibility score is determined by a combination of the expert selection frequency and the aggregated route score, specifically:

[0119]

[0120] in, Experts In clustering The The number of times a layer appears in the top-k route results. Experts The aggregated router score, These are weighting parameters used to balance the selection frequency and the importance of the route; Represents the experts in the l-th layer of cluster k. The trustworthiness score is obtained by weighting the selection frequency and the aggregated route score.

[0121] During the online phase, once a batch is formed from the bucket, the prediction module returns a batch-level trusted thermal expert prior based on the clustering identity of the batch. The expert scheduler combines the batch-level trusted thermal expert prior, the current GPU memory resident status, and the GPU-CPU resource status to allocate the GPU memory budget and determine the thermal expert fixing, expert prefetching, and expert replacement strategies.

[0122] Batch-level trusted hot expert priors are obtained by indexing the cluster identity of the batch in the offline clustering-hierarchical routing table. The routing table calculates the trusted popularity of each expert based on the historical expert selection frequency and aggregated route score, and records the Top-M hot expert set for each cluster and each MoE layer. The current GPU memory resident state and GPU-CPU runtime resource state do not change the prior itself, but are used to consume the prior online and determine memory budget, fixing, prefetching, and replacement actions.

[0123] Batch-level trusted hot expert priors are generated offline from historical routing logs and indexed online by the current batch cluster identity. In the offline phase, for each cluster, MoE layer, and expert, the number of times it is selected in the top-k routing results and its aggregation router score are counted. The popularity is calculated according to the aforementioned expert trusted popularity formula, and Top-M experts are selected to form a layer-by-layer hot expert set. In the online phase, this set is read based on the cluster identity after the batch is formed. The current parameter sub-block resident mask, available video memory, and GPU-CPU running status are only used to determine fixing, prefetching, and replacement actions and do not participate in the generation of the prior itself.

[0124] Specifically, cluster identity is used to find the corresponding layer-by-layer Top-M hot expert set in constant time, and historical expert selection frequency and aggregated route score are used to calculate the set offline according to the heat formula; the online scheduler then combines the resident mask of each expert parameter sub-block, available video memory and GPU-CPU execution status to determine which experts are fixed to resident, which experts are prefetched and which non-fixed expert sub-blocks need to be released.

[0125] In this embodiment, during the memory initialization phase, since the non-expert layer does not have selective activation features, model parameters other than those of experts are loaded into the GPU memory. First, the embedding module and the Layer 0 non-expert components are loaded, and the Layer 0 routing results are calculated based on these. Simultaneously, the remaining necessary components are loaded concurrently. After initialization, some globally trusted hot experts are prefetched into the GPU memory based on the global hot expert list and historical expert coverage information to reduce expert loading pauses during the initial inference phase and to provide an initial resident state for subsequent batch feature optimization.

[0126] Step 302: Obtain the layer-by-layer hot expert set from the offline routing statistics. The expert scheduler marks the experts in the layer-by-layer hot expert set as fixed experts.

[0127] For the current batch, the set of layer-by-layer thermal experts is read from the offline clustering-hierarchical routing table based on its clustering identity. The expert scheduler only marks experts in the set as fixed experts, ensuring their major parameter blocks remain in GPU memory and are not evicted. The set of fixed experts is:

[0128]

[0129] in, This represents the set of fixed experts comprised of fixed experts from each MoE layer; This represents the union of the thermal expert sets corresponding to all MoE layers l.

[0130] When the batch category changes, the expert scheduler updates the fixed expert set. This includes: fixing newly identified thermal experts and prefetching their parameter blocks into GPU memory; removing experts from the fixed state and placing them in the eviction pool; and executing thermal expert preloading as a low-priority background task. When a compute path generates a critical prefetch request, the compute path request is responded to first, and then the background preloading task is resumed.

[0131] Step 303: Combine the expert resident status and GPU-CPU resource status to perform expert resident and prefetch scheduling in GPU memory, which also includes a hot expert fixing strategy.

[0132] Step 303 specifically includes: reading the hierarchical thermal expert set based on batch clustering identity. , will satisfy (l,e)∈ Experts are marked as fixed experts, ensuring that their three main parameter blocks (gate_proj, up_proj, and down_proj) remain resident in GPU memory and are not evicted; when the batch category changes, new entries are prefetched. The experts have cancelled their withdrawal. The experts are fixed in state and added to the eviction pool; hot expert preloading is executed as a low-priority background task. When a real route generates a critical path prefetch request, background preloading is paused and the missing parameter sub-blocks required by the current layer are loaded first. Given the memory budget, the number of fixed experts, the prefetch queue prefix, and the number of bytes to be released are determined based on the current resident mask and available memory. When memory is insufficient, the least recently used mechanism of the sub-block level protected by batch fixed experts is invoked to release the parameter sub-blocks of non-fixed experts. This process only adjusts the resident and migration order of expert parameters and does not change the model routing results or weighted aggregation semantics.

[0133] Step 4: Based on the requirements of expert sub-blocks and the resident mask, a sub-block-level partial replacement mechanism is adopted for expert parameters, and only the expert parameter sub-blocks required for the current inference critical path are migrated between GPU memory and host memory.

[0134] In terms of expert parameter management, this invention divides expert parameters into three memory-dominant sub-blocks: gate_proj, up_proj, and down_proj, and records the resident status of these sub-blocks using a mask. When GPU memory is insufficient, the system no longer evicts experts on a complete basis. Instead, based on an LRU sliding window, combined with the fixed state of trusted hot experts and the sub-block resident mask, it performs partial sub-block evictions only in the expert set with a low probability of short-term reuse. This reduces redundant transmission caused by migrating complete experts and minimizes I / O waste caused by historical prediction errors.

[0135] Specifically:

[0136] Step 401: For each expert module, manage the expert parameters according to the sub-block granularity. Divide the expert parameters into three memory-dominant sub-blocks: gate_proj, up_proj, and down_proj, and record the resident status of the sub-blocks using a mask.

[0137] For each expert module This approach manages expert parameters at the sub-block level, rather than treating the entire expert as an indivisible replacement unit. The three main weight blocks in the expert that consume the most GPU memory are denoted as gate_proj, up_proj, and down_proj, and represented as follows: , and Lightweight parameters, due to their small memory usage, are not involved in eviction decisions.

[0138] Expert Module The resident state in GPU memory is encoded using a three-bit mask, specifically:

[0139]

[0140] in, Representation of expert modules The resident status of the three main parameter blocks in GPU memory; These represent the resident bits corresponding to the gate_proj, up_proj, and down_proj parameter blocks, respectively.

[0141] In this context, a bit value of 1 indicates that the corresponding parameter block is already residing in GPU memory, while a bit value of 0 indicates that the corresponding parameter block is not residing in GPU memory. For example, This indicates that the experts will remain on-site. This indicates that only the up_proj parameter block resides.

[0142] Step 402: Implement sub-block-level partial replacement through the Batch-Pinned Partial LRU (Batch-Pinned Expert-Protected Least Recently Used Sub-block-Level Partial) mechanism;

[0143] set up A fixed group of experts who will not participate in the expulsion. A set of non-fixed experts that can be expelled. Maintain an LRU queue The front of the queue consists of the least recently used experts, and the rear of the queue consists of the most recently used experts. The size is defined as... The sliding window is an LRU queue. The front of the middle The experts are:

[0144]

[0145] in, This represents a sliding candidate window consisting of the first W experts at the head of the LRU queue Q; This represents the operation of extracting the first W elements from the head of queue Q.

[0146] This sliding window represents a group of candidate experts with a low probability of reuse in the short term. Let the parameter block eviction order be... for and maintain the expulsion pointer When GPU memory needs to be reclaimed, it only occurs in the sliding window. Perform an expulsion operation: Start scanning for experts from the front of the team; if an expert... and Then the corresponding parameter block is evicted from the GPU memory. and will Set to 0. The scanning process continues until the released video memory reaches the required number of bytes. Until the sliding window has been completely scanned. Then the pointer is evicted. according to Move to the next block type.

[0147] When an expert is hit by the route again, if there are missing parameter blocks, the missing parameter blocks are prefetched as needed before execution, and the expert is moved to the tail of the LRU queue.

[0148] Through the aforementioned sub-block level partial replacement mechanism, the routing trust feature and the memory resident mask jointly determine the parameter sub-blocks that need to be migrated, which can avoid redundant transmission caused by complete expert replacement and reduce I / O waste caused by inaccurate expert prediction.

[0149] Step 5: Based on the actual routing results, perform online correction on the batch-level trusted scheduling features (i.e., batch-level scheduling information consisting of batch cluster identity, batch-level trusted hot expert priors, and current expert residency status), construct expert-level tasks, and adopt a ready-first out-of-order expert execution strategy to prioritize the execution of GPU-resident experts and high-demand experts.

[0150] During execution at the MoE layer, the system constructs expert-level tasks based on the actual routing results. The actual routing results are used to verify the aforementioned batch-level trusted scheduling characteristics: the GPU prioritizes executing already resident expert tasks and performs limited prefetching on non-resident experts with high hit counts; for long-tail cold experts with low hit counts, the CPU directly uses the weights in the host memory to perform forward computation to avoid loading the complete expert into the GPU memory for a small number of tokens; for experts with routing scores below a threshold, the system skips their computation to reduce the additional computation and transmission overhead caused by low-contribution experts.

[0151] Based on the actual routing results, expert-level tasks are constructed, and an out-of-order expert execution strategy is adopted to prioritize the execution of GPU-resident experts and high-demand experts, as detailed below:

[0152] In each layer of inference, an expert-level task group is constructed based on the actual routing results of the router. Let... Indicates entering the first If the expert set is already resident in the GPU memory at the time of layering, then all experts will be executed first. The relevant tasks enable the GPU to start computing immediately, reducing idle time caused by waiting for cold experts to load.

[0153] For the remaining non-resident experts, statistics are as follows: Layered experts The number of tokens allocated is denoted as and in accordance with Construct expert queues from largest to smallest. The expert scheduler prioritizes... The top few experts with a high hit count initiate GPU prefetching, allowing high-demand experts to be loaded into GPU memory as early as possible.

[0154] The aforementioned out-of-order execution only changes the execution order of expert tasks within the same MoE layer, without altering the routing results or the weighted aggregation semantics of expert outputs. A synchronization barrier is set before merging expert outputs to ensure that all selected expert outputs that have not been skipped have been completed. In other words, trusted feature optimization operates on the system scheduling path without changing the expert selection rules or model structure of the MoE model itself.

[0155] Step 6: Process cold experts and long-tail experts through GPU-CPU collaborative execution mechanism, and complete the MoE layer inference calculation under the recommendation quality constraint by combining score threshold skipping and CPU early stopping strategy, and output recommendation results.

[0156] During GPU-CPU collaborative execution, the system compares the coarse-grained execution latency of experts on the GPU and CPU paths. When the CPU first discovers that GPU execution is superior while scanning forward from the long-tail queue, the CPU stops processing subsequent experts and hands over the remaining tasks to the GPU path. The GPU and CPU paths synchronize before the expert output merging point at each layer to ensure that all selected expert outputs that have not been skipped have been completed. Except for the score threshold skipping strategy, which ignores low-contribution experts according to a preset threshold, out-of-order execution and GPU-CPU collaborative scheduling only change the execution order of expert tasks within the same layer, without changing the MoE routing results or the weighted aggregation semantics of the outputs of experts that have not been skipped.

[0157] The GPU-CPU collaborative execution mechanism handles cold experts and long-tail experts, and combines score threshold skipping and CPU early stopping strategies to complete the MoE layer inference computation, outputting the inference results as follows:

[0158] For experts who are not currently residing in the queue but have high demand, the GPU will retrieve them from the expert queue. The process begins with the head of the queue, loading its parameter block into GPU memory before execution to amortize the loading cost; for long-tailed cold experts that are not resident and have a small hit count, the CPU loads them from... Processing begins at the tail of the queue, using CPU-side resident weights to complete forward propagation, thus avoiding expensive GPU parameter loading and memory replacement for a small number of tokens.

[0159] The GPU and CPU execute their respective assigned expert tasks in parallel and synchronize at the output merging point. To prevent the CPU execution path from prolonging the completion time of the current layer, two strategies are adopted: score threshold skipping and early CPU stopping. This allows the reliability of the routing score and the GPU-CPU execution cost to jointly constrain the processing range of long-tail cold experts.

[0160] The score threshold skipping strategy is as follows: for experts assigned to CPU processing... Use its aggregated router scores Measure the expert's contribution to the final output. When At that time, skip the expert's calculations, where, The score threshold determined offline can be configured according to the accuracy target.

[0161] The CPU's early stopping strategy is as follows: when the remaining expert tasks are more suitable for GPU processing, the CPU stops consuming tasks and hands over the remaining tasks to the GPU. Specifically, this involves estimating the expert tasks separately. The processing latency on the GPU and CPU is calculated using the following formula:

[0162]

[0163]

[0164] in, Indicates that experts The estimated time for loading parameters into GPU memory. and They represent experts respectively. Processing the forward time of a single token on both GPU and CPU This represents the additional overhead of CPU execution. Because the CPU... Scan from the tail of the queue to the head of the queue, when the first condition is met... When this occurs, it indicates that the remaining expert tasks are more suitable for GPU processing, so the CPU stops executing and hands over the remaining expert tasks to the GPU. This represents the number of tokens retained and allocated to the l-th level expert e after the score threshold filtering. This represents the estimated total latency for expert e to complete parameter loading and forward computation on the GPU; This represents the estimated total latency after expert e completes the forward computation on the CPU and accounts for additional overhead.

[0165] Furthermore, the effectiveness of route-aware batch processing is measured by in-batch expert diversity and expert reuse rate, as follows:

[0166] For batch and MoE layer set ,set up Indicates a request In the layer The set of experts active in the evaluation routing window, then the batch The average number of activated experts is defined as:

[0167]

[0168] in, This represents the average number of activated experts in batch B at the examined MoE layer, which is the average number of different experts jointly activated by all requests within the batch at each layer. The smaller the value, the fewer different experts need to be stationed or loaded on each floor in the same batch.

[0169] batch The expert reuse rate is defined as:

[0170]

[0171] in, This represents the average expert reuse rate of batch B at the examined MoE layer, which is the average ratio of the total number of request-level expert activations to the number of different experts jointly activated. The larger the value, the more expert visits each activated expert serves on average within the same batch, and the better the expert reuse effect.

[0172] In this embodiment, the method is applicable to scenarios where GPU memory cannot fully accommodate all MoE expert parameters, and where a trusted recommendation model service needs to be deployed on local devices, edge devices, and resource-constrained inference servers.

[0173] like Figure 2 As shown, the left side represents the expert activation heat of each batch without route-aware clustering, and the right side represents the expert activation heat of each batch after clustering according to early layer route affinity. The horizontal axis represents the MoE layer, the vertical axis represents the expert number, and the color intensity represents the frequency with which an expert is requested for activation within the corresponding layer. As the batch size increases from 4 to 16, the set of activated experts in the unclustered batches becomes significantly more dispersed, and the number of cold experts within the batch increases. After clustering, the activation distribution is concentrated in a smaller number of experts, indicating that route-homogeneous batches can reduce the diversity of experts within the batch, improve the expert reuse rate, and reduce the exchange of experts between host memory and GPU memory.

[0174] like Figure 3As shown, request pairs are first divided into high-similarity and low-similarity groups based solely on the route similarity S0 during the Layer 0 pre-filling stage. Then, the expert activation similarity for each subsequent MoE layer is calculated using the complete pre-filling and decoded route logs. In GPT-OSS, the S0 of the high-similarity group is approximately 0.90, with subsequent layers showing similarities above 0.7 and an inter-layer average of approximately 0.8. The S0 of the low-similarity group is approximately 0.33, with subsequent layers averaging approximately 0.23. DeepSeek-MoE exhibits the same trend. This indicates that requests with similar activation expert sets at Layer 0 maintain similar routing trajectories at deeper levels, and early-layer route signatures can serve as reliable proxies for the full model's expert access patterns.

[0175] like Figure 4 As shown, the offline tracing link records historical request identifiers, tokens, MoE layers, selected experts, and corresponding router scores by setting hooks at the MoE Gate. Based on this, the system generates layer 0 trusted route signatures, cluster centroids, and expert heat tables organized by cluster and layer. The online batch processing link first performs embedding and layer 0 lightweight gate probes on new requests. Without executing expert FFN, it obtains the top-k expert set for each token, calculates the trusted route signature, matches the nearest cluster centroid, and adds the request to the corresponding bucket queue. When the number of requests in the bucket reaches the target batch size or the waiting time limit, a route homogeneous batch is formed, and the layer-by-layer hot expert set of this batch is retrieved based on the cluster identity for subsequent fixing, prefetching, and replacement scheduling.

[0176] like Figure 5 As shown, the MoE model can be decomposed into modules such as Embedding, Decoder Stack, Final Normalization Layer, and LM Head. Non-expert components do not have selective activation features and can reside in GPU memory during the initialization phase. Each expert module in the MoE layer is further composed of sub-modules such as gate_proj, up_proj, down_proj, and lightweight activation functions. gate_proj, up_proj, and down_proj occupy the main GPU memory space for expert parameters, thus serving as parameter sub-blocks that can be independently resident, prefetched, and evicted, respectively. Components with minimal footprint, such as activation functions, are not included in the eviction decision. This decomposition allows the system to migrate missing sub-blocks on demand, without having to transfer the complete expert every time.

[0177] like Figure 6As shown, the priority expert replacement strategy treats experts as indivisible units. When memory is insufficient, it evicts all parameter blocks of the same expert's `gate_proj`, `up_proj`, and `down_proj` at once. Although this can quickly free up a large amount of space, when the expert is hit again, it needs to be completely refilled, which can easily lead to transmission amplification. The sequential sub-block replacement strategy, on the other hand, evicts parameter blocks in a fixed order of `gate_proj`, `up_proj`, and `down_proj` within the sliding window of the least recently used queue. It stops immediately after the required space is freed, and retains the remaining sub-blocks of the expert that have not yet been evicted. When the expert is hit again, it only fills in the missing sub-blocks. By limiting the ordered eviction to a small candidate window, it can avoid a large number of experts being in a fragmented partially resident state at the same time and reduce redundant PCIe transmissions.

[0178] like Figure 7 As shown, after obtaining the actual routing results, the system first performs token-expert task filtering based on routing scores, and then constructs an expert-level task queue. Experts already residing in GPU memory are immediately executed by the GPU; high-demand experts that are not residing but have a large number of hit tokens are taken from the head of the queue and asynchronously prefetched to the GPU for execution; long-tail cold experts with a small number of hits are executed directly by the CPU from the tail of the queue using the weights in the host memory; low-contribution tasks below the score threshold are skipped. The GPU, I / O threads, and CPU proceed in parallel and are synchronized at the output merging point in the MoE layer; when the GPU estimated latency of the remaining tasks is not higher than the CPU estimated latency, the CPU stops early and returns the remaining tasks to the GPU, thereby avoiding the long tail of the CPU extending the overall layer completion time.

[0179] This embodiment takes a large-scale hybrid expert model trusted recommendation reasoning service under a memory-constrained environment as an example to verify the GPU-CPU heterogeneous collaborative reasoning method for efficient and trusted recommendation proposed in this invention.

[0180] This embodiment uses DeepSeek-MoE-16B-Base, DeepSeek-MoE-16B-Chat, and Mixtral-8×7B as experimental models. DeepSeek-MoE-16B-Base and Mixtral-8×7B are used for throughput and I / O performance verification, while DeepSeek-MoE-16B-Chat is used for model quality verification.

[0181] In this embodiment, offline route clustering uses the Train partition of the WikiText-2 dataset to collect route logs, while online evaluation uses the Test partition of the WikiText-2 dataset. For MMLU model quality evaluation, the top 30% of samples for each subject are used to generate offline route statistics, and the remaining samples are used for testing. For clustering settings, WikiText-2 requests are divided into 8 clusters, and the number of clusters in the MMLU task is consistent with the number of subjects. These settings are used to verify whether trusted route features remain effective between offline statistics and online requests, rather than simply verifying the effect of static heat caching.

[0182] Table 1 Experimental Platform Configuration

[0183]

[0184] To reduce the control complexity of dynamic expert loading and partially resident states, this embodiment sets an expert placeholder for each expert, allowing the GPU inference thread to access expert parameters through a unified interface. When an expert or its parameter sub-blocks are not resident in GPU memory, the scheduler initiates asynchronous transmission through a dedicated I / O thread; when GPU memory is reclaimed, the evicted block is directly overwritten to free up space, without copying it back to host memory, thereby avoiding unnecessary data transfer.

[0185] Table 2 Hyperparameter Configuration

[0186]

[0187] Experiment Result 1: The impact of route-aware batch processing on expert diversity and reusability;

[0188] This embodiment compares the expert diversity and reusability of random batch processing, length-aware batch processing, and the method of this invention on the DeepSeek-MoE model on the RTX 5090 platform. For each request r_i and MoE layer l, the set of experts activated in the measurement window is denoted as E_l(r_i). For batch B, the average number of activated experts per layer D(B) is defined as:

[0189]

[0190] Where L is the set of MoE layers being evaluated, and the smaller D(B) is, the fewer different experts need to be maintained or loaded for each layer.

[0191] The expert reuse rate R(B) is defined as:

[0192]

[0193] A larger R(B) indicates that each activated expert is accessed and reused by more tokens within the same batch.

[0194] Where B represents the batch to be evaluated, L represents the set of MoE layers to be evaluated, |L| represents the number of MoE layers evaluated, r_i represents the i-th request in batch B, E_l(r_i) represents the set of experts activated by request r_i within the measurement window of the l-th layer, ∪_{r_i∈B}E_l(r_i) represents the set of different experts jointly activated by all requests in the batch at the l-th layer, |·| represents the cardinality of the set; D(B) represents the average number of experts activated per layer in batch B; R(B) represents the average expert reuse rate in batch B, the numerator of which is the sum of the number of experts activated by each request at that layer, and the denominator is the number of different experts jointly activated at that layer.

[0195] Table 3. Average number of experts activated per layer for different batch processing strategies on RTX 5090

[0196]

[0197] Table 4 Expert reuse rate for different batch processing strategies on RTX 5090

[0198]

[0199] As shown in Tables 3 and 4, compared to random batch processing, the method of the present invention reduces the average number of activated experts per layer by 23.0%, 21.5%, and 17.3% when batch sizes are 8, 16, and 32, respectively; while increasing the expert reuse rate by 29.4%, 27.4%, and 20.8%, respectively. Compared to length-aware batch processing, the method of the present invention can still significantly reduce the number of activated experts and improve the expert reuse rate. This result indicates that request length is not a reliable proxy for MoE routing behavior. By utilizing early layer routing, historical coverage, and Expert-IDF weights to form trusted routing features, the expert reuse relationship between recommendation requests can be captured more directly.

[0200] Experimental Result 2: The impact of partial replacement mechanism on I / O cost;

[0201] This embodiment further compares the I / O costs of full expert replacement versus partial replacement on the RTX 5090 platform. The full expert replacement method transmits the entire expert when a non-resident expert is required, while the method of this invention only transmits the actually needed memory usage dominant parameter sub-block.

[0202] Table 5. I / O Costs of Full Expert Replacement and Partial Replacement on RTX 5090

[0203]

[0204] As shown in Table 5, compared with whole expert replacement, the partial replacement mechanism of this invention reduces I / O costs by 44.3%, 46.9%, and 48.8% for batch sizes of 8, 16, and 32, respectively. As the batch size increases, more cold experts are exposed, making it easier for whole expert replacement to transmit non-critical path parameters. However, this invention utilizes trusted hot expert profiles and sub-block residency masks to identify the actual necessary parameter blocks, migrating only the necessary sub-blocks, thereby improving the utilization efficiency of GPU memory and PCIe bandwidth and reducing the tail latency risk of trusted recommendation inference.

[0205] Experimental Results 3: End-to-end throughput, I / O traffic, and model quality;

[0206] This embodiment selects Hugging Face Accelerate (Hugging Face open-source large model training and inference acceleration framework), MoE-Infinity (expert offloading and prefetching framework for hybrid expert models), and Fiddler (a hybrid expert model inference framework based on CPU-GPU orchestration) as comparison methods. Accelerate is a general LLM offloading framework that moves parameters between CPU and GPU at the layer granularity; MoE-Infinity is a MoE-specific prefetching framework that typically performs replacement at the whole expert granularity; Fiddler divides hot and cold experts offline and selects CPU or GPU for execution at runtime based on the cost model. To ensure a fair comparison, this embodiment replaces Fiddler's serial CPU-GPU execution with a parallel implementation while retaining its original algorithm logic.

[0207] Experimental results show that on the DeepSeek-MoE-16B-Base and Mixtral-8×7B models, the method of this invention achieves the highest throughput across all runnable configurations. Compared to Fiddler, the throughput of the method of this invention is improved by 27% to 88%. Compared to Accelerate, the method of this invention reduces the average data transfer volume of DRAM-VRAM by 55% to 83%; compared to MoE-Infinity, the method of this invention reduces the average data transfer volume of DRAM-VRAM by 50% to 53%. These results demonstrate that the trustworthy scheduling features for trustworthy recommendation reasoning construction can simultaneously improve batch formation, expert residency, and parameter migration, rather than only achieving localized benefits in a single stage.

[0208] Analysis of GPU and CPU utilization reveals that while Accelerate and MoE-Infinity maintain high GPU utilization, their throughput is low, primarily due to frequent I / O blocking computation progress. Fiddler exhibits low GPU utilization and high CPU utilization, indicating that its performance gains mainly come from CPU-side execution rather than effective GPU-CPU collaboration. When the number of CPU cores is limited, Fiddler's performance drops significantly, while the method of this invention maintains relatively stable throughput, demonstrating that this method can more reasonably delineate the boundaries between GPU and CPU tasks by utilizing reliable features comprised of routing scores, expert hit scale, and execution latency.

[0209] Regarding model quality, this embodiment employs the WikiText-2 perplexity and MMLU accuracy evaluation score threshold skipping strategy. Experimental results show that score threshold skipping only causes slight quality fluctuations, and both model perplexity and accuracy remain within acceptable ranges. This indicates that the credible feature optimization in this invention mainly functions in expert scheduling, parameter migration, and cold expert processing paths, reducing computational and transmission overhead while maintaining relatively stable model output quality.

[0210] Experimental Result 4: Ablation Verification of Each Functional Module;

[0211] This embodiment further conducts ablation experiments to verify the independent function of each component module. When route-aware batch processing is replaced with random batch processing, the average throughput on RTX 4060Ti and RTX 5090 decreases by 17.8% and 20.1%, respectively; when partial replacement is disabled, the average throughput on the two GPUs decreases by 11.6% and 8.0%, respectively; when out-of-order execution is removed, the throughput on the two GPUs decreases by 20.9% and 18.7%, respectively; when CPU participation is disabled, the average throughput on the two GPUs decreases by 37.4% and 30.2%, respectively.

[0212] The above results demonstrate that the performance improvement of this invention does not stem from a single scheduling rule, but rather from the combined effects of trusted route feature construction, route-aware batch processing, sub-block-level partial replacement, ready-first out-of-order execution, and GPU-CPU collaborative tail processing. Specifically, trusted route feature construction transforms the MoE recommendation inference state into a stable basis for scheduling, while CPU-side long-tail cold expert processing prevents cold experts from blocking GPU computation progress. Together, they improve service efficiency in memory-constrained scenarios.

[0213] In summary, this embodiment verifies that the GPU-CPU heterogeneous collaborative inference method for efficient and reliable recommendation using a memory-constrained hybrid expert model proposed in this invention can transform recommendation request routing, historical expert access, router scores, expert residency status, and GPU-CPU resource status into stable reliable scheduling features under limited GPU memory conditions. These features drive batch formation, hot expert residency, sub-block migration, and GPU-CPU task partitioning, significantly reducing intra-batch expert diversity, minimizing DRAM-VRAM data transfer, improving end-to-end inference throughput, and maintaining relatively stable recommendation quality. Therefore, this invention is applicable to reliable recommendation inference scenarios in local devices, edge platforms, and data centers with limited GPU memory.

[0214] Example 2

[0215] This embodiment provides a GPU-CPU heterogeneous collaborative inference system for memory-constrained hybrid expert models aimed at efficient and reliable recommendation. The system consists of a reliable scheduling feature construction module, a route-aware request batch processing module, an expert scheduling module, and a GPU-CPU collaborative inference module.

[0216] Trusted scheduling feature construction module: responsible for transforming early layer routing results, historical expert coverage information, expert routing scores, batch-level trusted hot expert priors (i.e., the Top-M hot expert set clustered by batch identity and MoE layer index), expert residency status and execution cost into unified inference scheduling features.

[0217] Route-aware request batch processing module: Generates route homogeneous batches based on the early-layer route affinity represented by trusted route signatures;

[0218] Expert scheduling module: performs expert fixing, pre-fetching, and sub-block-level replacement based on batch-level trusted thermal expert priors;

[0219] GPU-CPU collaborative reasoning module: Performs expert-level task scheduling based on the actual routing results, and maps resident experts, high-demand experts, and long-tail cold experts to appropriate computing resources respectively.

[0220] This invention unifies early layer routing affinity, historical request coverage, expert routing scores, batch-level trusted hot expert priors, expert residency status, and CPU / GPU execution costs into trusted scheduling features. These features are then used to drive request clustering, expert fixing, sub-block replacement, and GPU-CPU collaborative execution decisions, respectively. Simultaneously, the scheduling features are constrained online through real routing results, score thresholds, and CPU early stopping mechanisms to reduce the impact of noisy historical information and low-contribution expert computations on memory-constrained trusted recommendation inference services.

[0221] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models aimed at efficient and reliable recommendation, characterized in that: Includes the following steps: Step 1: Obtain the MoE inference request for the trusted recommendation task, and construct the trusted route signature of the request based on the early layer routing results, historical expert coverage information and expert routing scores. Step 2: Cluster route-aware requests based on the route affinity represented by the trusted route signature, and aggregate recommendation requests with similar expert access patterns and reusable value into route homogeneous batches. Step 3: Based on the cluster identity of the homogeneous batch of routes, read the batch-level trusted thermal expert prior from the offline clustering-hierarchical thermal expert statistics table, and combine it with the current GPU memory resident status and GPU-CPU runtime resource status to determine the memory budget of fixed thermal experts, expert prefetching and non-fixed experts, so as to provide a scheduling basis for subsequent sub-block level partial replacement. Step 4: Use a sub-block level partial replacement mechanism for expert parameters, and migrate only the expert parameter sub-blocks required for the current inference critical path between GPU memory and host memory; Step 5: Perform online correction of batch-level trusted scheduling characteristics based on actual routing results, construct expert-level tasks, and adopt a ready-first out-of-order expert execution strategy to prioritize the execution of GPU-resident experts and high-demand experts. Step 6: Process cold experts and long-tail experts through GPU-CPU collaborative execution mechanism, and complete the MoE layer inference calculation under the recommendation quality constraint by combining score threshold skipping and CPU early stopping strategy, and output recommendation results.

2. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: In step 1, the specific implementation details are as follows: Step 101: Execute a Layer 0 lightweight router probe on the incoming MoE inference request to obtain the top-k expert set corresponding to each token; Step 102: Construct the expert frequency vector: Summarize the routing results of MoE inference requests at layer 0 into an expert frequency vector; Step 103: Calculate the expert's request coverage number and smooth inverse coverage weight, and perform Expert-IDF reweighting and normalization on the expert frequency vector to obtain the trusted route signature.

3. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: In step 2, the specific implementation details are as follows: Route-aware request clustering uses trusted route signatures as the clustering basis to transform the MoE routing behavior of recommendation requests into batch-level expert reusable features, specifically: During the offline phase, the route signatures of historical requests are clustered into For each cluster, obtain the corresponding cluster centroid. ; During the online phase, for the arriving requests Based on its trusted route signature The cosine similarity with each cluster centroid is assigned to the nearest cluster centroid, specifically: ； in, This indicates the nearest cluster number to which request r is assigned; Indicates the candidate cluster number; This represents the trusted route signature obtained by reweighting and normalizing request r using Expert-IDF; Let represent the centroid of the k-th cluster; Then, the request Add to the corresponding bucket queue In the middle; when the bucket queue reaches the target batch size or the request waiting time reaches the preset waiting time boundary, the request is taken out from the bucket queue to form a homogeneous batch of routes.

4. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: In step 3, the specific implementation details are as follows: Step 301: For each cluster and each MoE layer, calculate the expert trust score based on the historical routing logs, and record the Top-M hot expert set corresponding to the cluster and the layer. Step 302: Obtain the layer-by-layer hot expert set from the offline routing statistics. The expert scheduler marks the experts in the layer-by-layer hot expert set as fixed experts. Step 303: The expert scheduler combines batch-level trusted thermal expert priors, the current GPU memory resident status, and the GPU-CPU resource status to allocate GPU memory budget, and determines the thermal expert fixing, expert prefetching, and expert replacement strategies.

5. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: In step 4, the specific implementation details are as follows: Step 401: For each expert module, manage the expert parameters according to the sub-block granularity. Divide the expert parameters into three memory-dominant sub-blocks: gate_proj, up_proj, and down_proj, and record the resident status of the sub-blocks using a mask. Step 402: Implement sub-block-level partial replacement using the Batch-Pinned Partial LRU mechanism.

6. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: In step 5, the specific implementation details are as follows: In each layer of inference, an expert-level task group is constructed based on the actual routing results of the router. Indicates entering the first A collection of experts that already resides in the GPU memory at the layer level; First, execute all procedures with experts. The relevant tasks enable the GPU to start computing immediately, reducing idle time caused by waiting for cold experts to load; For the remaining non-resident experts, statistics are as follows: Layered experts The number of tokens allocated is denoted as and in accordance with Construct expert queues from largest to smallest. ; Expert scheduler prioritizes The top few experts with a high hit count initiate GPU prefetching, allowing high-demand experts to be loaded into GPU memory as early as possible.

7. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: In step 6, the specific implementation details are as follows: For experts who are not currently residing in the queue but have high demand, the GPU will retrieve them from the expert queue. The process begins with the head of the queue, loading its parameter block into GPU memory before execution; for long-tailed cold experts that are not resident and have a small hit count, the CPU loads them from... Processing begins at the tail of the queue, and forward propagation is completed directly using the CPU-side resident weights. The GPU and CPU execute their respective assigned expert tasks in parallel and synchronize at the output merging point.

8. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: The score threshold skipping strategy is as follows: For experts assigned to CPU processing Use its aggregated router scores Measure the expert's contribution to the final output; when At that time, skip the expert's calculations, where, The score threshold determined offline can be configured according to the accuracy target.

9. The GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models oriented towards efficient and reliable recommendation, as described in claim 1, is characterized in that: The CPU early termination policy is as follows: When the remaining tasks are better suited to be handled by the GPU, the CPU stops consuming tasks and hands over the remaining tasks to the GPU. Specifically, experts were estimated separately. The processing latency on the GPU and CPU is calculated using the following formula: ； ； in, Indicates that experts The estimated time for loading parameters into GPU memory. and They represent experts respectively. Processing the forward time of a single token on both GPU and CPU This indicates the additional overhead of CPU execution; This represents the number of tokens retained and allocated to the l-th level expert e after the score threshold filtering. This represents the estimated total latency for expert e to complete parameter loading and forward computation on the GPU; This represents the estimated total latency after expert e completes the forward computation on the CPU and accounts for additional overhead.

10. A GPU-CPU heterogeneous collaborative inference system for memory-constrained hybrid expert models aimed at efficient and reliable recommendation, implementing the GPU-CPU heterogeneous collaborative inference method for memory-constrained hybrid expert models aimed at efficient and reliable recommendation as described in any one of claims 1 to 9, characterized in that: The system includes: The system includes a trusted scheduling feature construction module, a route-aware request batch processing module, an expert scheduling module, and a GPU-CPU collaborative inference module. Trusted scheduling feature construction module: responsible for transforming early layer routing results, historical coverage, routing scores, batch-level trusted hot expert priors, expert residency status, and execution costs into unified inference scheduling features; Route-aware request batch processing module: Generates route homogeneous batches based on the early-layer route affinity represented by trusted route signatures; Expert scheduling module: performs expert fixing, pre-fetching, and sub-block-level replacement based on batch-level trusted thermal expert priors; GPU-CPU collaborative reasoning module: Performs expert-level task scheduling based on the actual routing results, and maps resident experts, high-demand experts, and long-tail cold experts to appropriate computing resources respectively.

Images