Heterogeneous inference acceleration method and system for mixed expert models

Abstract

The application provides a heterogeneous reasoning acceleration method and system of mixed expert models, comprising: based on a representative data set of a multi-task context, analyzing the historical activation behavior of each expert, constructing an expert portrait representing the correlation between the expert and the task, and providing prior knowledge for scheduling; a cost prediction model is established for a heterogeneous computing unit, and the time delay of different execution paths is predicted according to the runtime input characteristics (such as data volume); during the reasoning process, the expert portrait and the cost prediction model are combined to select the optimal path for the activated expert from the candidate paths (CPU execution, GPU direct execution, weight migration to GPU execution), control task execution and data migration; based on real-time expert activation statistics and system state, dynamically update the expert portrait and the expert's residence strategy in the heterogeneous environment, and form a feedback loop. The application realizes adaptive optimization of the expert reasoning path, and solves the performance degradation problem of the traditional static allocation strategy in the complex input scene.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence inference optimization technology, specifically to a method and system for accelerating heterogeneous inference using hybrid expert models. More particularly, it relates to a method and system for accelerating inference using hybrid expert models based on input feature perception and dynamic weight transfer. Background Technology

[0002] In recent years, large-scale language models represented by the Hybrid Expert (MoE) architecture have been widely adopted. This architecture, through a sparse activation mechanism, calls only a small subset of expert subnetworks when processing each input, thus effectively controlling computational load while expanding model capacity. However, in practical deployments, especially in heterogeneous computing environments with limited resources such as graphics processing unit (GPU) memory, the inference efficiency of MoE models faces significant bottlenecks.

[0003] Existing systems typically employ fixed expert partitioning or simple hot expert resident mechanisms, failing to dynamically adjust based on the input task domain or semantic features. This leads to wasted GPU resources or loading delays during cross-domain inference. In resource-constrained environments (such as single-machine or edge-based systems), GPU memory is insufficient to simultaneously accommodate all expert weights, and frequent data migration between CPU and GPU increases transmission latency. Traditional scheduling strategies rely on fixed thresholds or heuristic judgments, neglecting the impact of dynamic factors such as the number of input tokens, sparsity, and batch size on computational latency.

[0004] Therefore, there is a need for an efficient MoE inference method that can adaptively allocate experts among heterogeneous devices, dynamically predict latency based on input, and migrate weights in real time. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a heterogeneous inference acceleration method and system for hybrid expert models in resource-constrained environments that can solve the memory bottleneck and latency fluctuation problems of hybrid expert (MoE) architectures during inference in resource-constrained environments.

[0006] A heterogeneous inference acceleration method based on a hybrid expert model, provided by the present invention, is applied in a heterogeneous computing environment containing a first computing unit and a second computing unit, comprising: Offline expert characteristic analysis steps: Before model inference, based on a representative dataset covering different task contexts, analyze the historical activation behavior of each expert in the hybrid expert model; by statistically analyzing and processing the data of the behavior, establish expert profile data that can characterize the relationship between experts and task context; The cost prediction step involves establishing a cost prediction model for the first and second computing units in the heterogeneous computing environment. The cost prediction model is used to predict the cost of performing a computing task in the first computing unit and the cost of transferring expert weights from the first computing unit to the second computing unit and having the second computing unit perform the computing task, based on the runtime input features during model inference. Dynamic scheduling execution steps: During model inference, when an expert is activated by the gating network, based on pre-established expert profile data and real-time running status, and using the cost prediction model, an optimal execution path is dynamically selected from multiple candidate execution paths; the candidate execution paths include at least: execution on the first computing unit, execution after migrating the expert weights to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing tasks and data migration operations according to the selected optimal path; Online adaptive adjustment steps: During model inference, continuously monitor and statistically analyze real-time expert activation information and / or system status; dynamically update expert profile data and / or adjust expert residency strategies in heterogeneous computing environments based on the monitored real-time feedback. Preferably, in the offline expert feature analysis step, the representative dataset includes multiple domains in natural language processing tasks, such as mathematical calculation, code generation, and common sense reasoning. By running a hybrid expert model for inference and using an analyzer to record the frequency and probability of activation of each expert at each layer, a domain-expert heat matrix is ​​constructed as an expert profile library. Preferably, in the execution cost prediction step, the cost includes time delay; The runtime input characteristics include the amount of data to be processed, the data type, and the complexity of the task. The first computing unit is a CPU, and the second computing unit is a single GPU.

[0007] Preferably, for the CPU computing unit, a linear fitting model is used to establish the computing latency model, as shown in the following formula:

[0008] In the formula, α and β are fitting parameters, and s is the number of tokens that the expert needs to process; For GPU computing units, a constant-fit model is used to establish the computational latency, as shown in the following formula:

[0009] In the formula, γ is a constant; Based on the measured PCIe bandwidth B between the CPU and GPU, the weight transfer delay and activation round-trip delay are established, with the following formulas:

[0010]

[0011] Where d is the hidden dimension, This indicates a delay in weight transfer. Indicates activation round-trip delay. Indicates the weight of the experts. Indicates the size of the data type. Preferably, in the dynamic scheduling execution step, when an expert is activated, if the expert weight has already resided in the cache of the second computing unit, then the execution is performed directly in the second computing unit. If the expert weights are in the memory of the first computing unit, then the cost prediction model is used to compare the cost of migrating the weights to the second computing unit for execution. The cost of activating the transfer to the first computing unit for execution. Choose the path with lower cost to execute, as shown in the following formula:

[0012]

[0013] In the formula, This indicates a delay in weight transfer. This indicates the activation round-trip delay.

[0014] Preferably, in the online adaptive adjustment step, the recent activation frequency of each expert is statistically analyzed using a sliding window as the short-term popularity, and then weighted and fused with the long-term popularity in the offline profile database to obtain a comprehensive popularity score. When the second computing unit runs out of memory, the expert with the lowest score is ousted using an LRU or similar strategy based on the overall popularity score.

[0015] Preferably, when the online statistical results deviate too much from the offline profiles, the similarity between the current activation distribution and the profiles of each domain is automatically calculated in each pre-filling stage, the best matching domain profile is reselected, and the long-term prior weights are adjusted accordingly.

[0016] A heterogeneous inference acceleration system based on a hybrid expert model, provided by the present invention, includes: Offline expert characteristic analysis module: Before model inference, based on a representative dataset covering different task contexts, the historical activation behavior of each expert in the hybrid expert model is analyzed; by statistically processing the data of the behavior, expert profile data that can characterize the relationship between experts and task context is established; Execution cost prediction module: For the first computing unit and the second computing unit in the heterogeneous computing environment, a cost prediction model is established; the cost prediction model is used to predict the cost of executing a computing task in the first computing unit and the cost of transferring expert weights from the first computing unit to the second computing unit and having the second computing unit execute the computing task, based on the runtime input features during model inference. Dynamic scheduling execution module: During model inference, when an expert is activated by the gating network, based on pre-established expert profile data and real-time running status, and using the cost prediction model, the system dynamically selects an optimal execution path from multiple candidate execution paths; the candidate execution paths include at least: execution on the first computing unit, execution after migrating the expert weights to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing tasks and data migration operations according to the selected optimal path; Online adaptive adjustment module: During model inference, continuously monitors and statistically analyzes real-time expert activation information and / or system status; dynamically updates expert profile data and / or adjusts expert residency strategies in heterogeneous computing environments based on the monitored real-time feedback.

[0017] Preferably, in the offline expert feature analysis module, the representative dataset includes multiple domains in natural language processing tasks, such as mathematical calculation, code generation, and common sense reasoning. By running a hybrid expert model for inference and using an analyzer to record the frequency and probability of each expert being activated at each layer, a domain-expert heat matrix is ​​constructed as an expert profile library. In the execution cost prediction model, the cost includes time latency; the runtime input features include the amount of data to be processed, the data type, and the task complexity; the first computing unit is a CPU, and the second computing unit is a single GPU.

[0018] Preferably, in the online adaptive adjustment module, the recent activation frequency of each expert is statistically analyzed using a sliding window as the short-term popularity, and then weighted and fused with the long-term popularity in the offline profile database to obtain a comprehensive popularity score. When the second computing unit runs out of memory, the expert with the lowest score is ousted using an LRU or similar strategy based on the overall popularity score.

[0019] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention obtains a "domain-expert heat matrix" by performing expert activation statistics and cluster analysis on samples from different task domains (such as mathematics, code, translation, common sense reasoning, etc.), which solves the technical problem of unbalanced expert use and difficulty in effectively reusing GPU high-speed storage in existing static expert allocation in cross-domain scenarios.

[0020] 2. This invention fits the CPU / GPU computation latency curve based on input features such as token quantity, activation sparsity, and batch size, and integrates the weight transfer time of PCIe / NVLink transmission bandwidth. This solves the problem that existing heuristic threshold strategies cannot quantify the real cost of "transferring weights vs. transferring activations", making device selection predictable and interpretable.

[0021] 3. This invention compares the combined latency of "weight migration to GPU execution" and "activation transfer to CPU execution" during runtime, and adaptively determines the execution path and resident device of experts by combining offline profiling and online statistics, thus solving the problem of frequent loading jitter caused by the inefficiency of existing CPU / GPU collaboration and insufficient video memory.

[0022] 4. During the calculation of the previous layer / the previous iteration, the present invention prefetches the weights of experts that may be activated in the future, and uses a sliding window combined with long-term profiles to perform heat fusion, thereby reducing cold start overhead and improving expert cache hit rate, thereby reducing average response latency and tail latency.

[0023] 5. This invention significantly reduces GPU memory usage (only thermal experts reside for a long time), reduces PCIe transmission volume, and improves end-to-end throughput and stability through a dual-channel strategy of "domain prior + runtime feedback". It does not depend on model structure changes and has universality and portability for multiple MoE architectures and various hardware forms. Attached Figure Description

[0024] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall system architecture of the present invention.

[0025] Figure 2 This is a schematic diagram of offline expert profile generation in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the runtime expert migration and cache scheduling process in an embodiment of the present invention. Detailed Implementation

[0027] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0028] Example 1 This embodiment demonstrates an inference acceleration method for hybrid expert models in a heterogeneous computing environment that includes a CPU (first computing unit) and a single GPU (second computing unit). The method establishes a heat profile by offline analysis of the activation frequency of experts in different task domains (such as mathematics and code), establishes a linear / constant latency model based on the number of input tokens online, and combines it with a bandwidth model to predict the total latency. Finally, it dynamically decides whether the expert should be executed on the CPU or migrated to the GPU, and updates the expert heat profile through a sliding window to adjust the GPU cache.

[0029] According to the present invention, a hybrid expert model inference acceleration method is provided, such as... Figure 1 As shown, it includes the following steps: The offline expert profile generation process involves: collecting datasets from different task domains, recording the activation feature information of each expert during the reasoning process, and constructing a domain-expert heat matrix. The activation features include not only the frequency of expert selection but also statistical quantities such as average gating probability, computational load, or output energy to reflect the expert's overall activity level within the domain. Clustering or dimensionality reduction algorithms are used to analyze the activation feature information to obtain the activity weight of each expert in different domains, generating an offline expert profile library as a priori basis for runtime expert allocation. The datasets from different task domains include representative task domain datasets such as mathematics, code, translation, and commonsense reasoning. Figure 2 As shown, the offline expert profile generation steps include: Step S1.1: Collect datasets from different task domains and preprocess the data in the datasets. The preprocessing includes unifying the text length distribution, tokenizer and decoding strategies, removing abnormal and duplicate samples, and ensuring consistent statistical standards.

[0030] Step S1.2: Perform inference on the hybrid experts and record the activation count and probability distribution of each expert at each layer. Without changing the model structure and weights, perform inference on each sample, record the expert selection results and their probabilities (top-k index and weight) of each layer's gating output, and generate hierarchical expert activation vectors by aggregating by token / sample.

[0031] Step S1.3: Perform model inference on the dataset for each task domain, and count the number of times each expert at each layer is selected by the gating module within that domain. In each forward computation, the gating layer outputs the score vector of all experts, and the system records the scores of the top experts. k The expert with the highest score is selected, and their activation count is accumulated.

[0032] For the statistical results of all samples in domain d, let the total number of times expert e is activated be denoted as . The total number of tokens processed was The average activation frequency of expert e in domain d is defined as:

[0033] All layers, all experts The values ​​are organized into a "domain-expert activation frequency matrix" to represent the usage frequency of each expert in different domains.

[0034] Optionally, principal component analysis (PCA) or clustering operations can be performed on the matrix data to identify expert groups with similar activation distributions, thereby forming a hierarchical expert heat table to provide a basis for subsequent expert placement and heterogeneous mapping.

[0035] Step S1.4: Identify the expert group structure using inter-layer clustering algorithms (such as K-means, DBSCAN, or PCA dimensionality reduction) to form an expert profile library; bind the expert heat table with the model weight index to support subsequent runtime loading optimization. Bind the heat matrix with the storage index of expert weights to form an expert profile library, providing a priori list for runtime hot expert residency and candidate prefetching.

[0036] Comprehensive latency prediction steps: During model runtime, based on input features such as input sequence length, token activation count, batch size, and task type, the actual execution time of experts on CPU and GPU is collected, and a functional relationship between input size and latency is fitted. The weighted transmission latency curve is measured based on the data bandwidth of PCIe or NVLink, and combined with the calculated latency curve to construct a comprehensive latency prediction model for subsequent dynamic scheduling decisions. The comprehensive latency prediction steps include: Step S2.1: Feature Definition and Sampling. Define input scale features, including sequence length, number of activated tokens s in the current iteration, batch size, beam width, sparsity index, etc.; sample and run for several rounds with different values.

[0037] Step S2.2: CPU / GPU calculate latency fitting, the formulas are as follows: The CPU latency model uses linear fitting: ; The GPU latency model uses a constant-fit form: ; Where s is the number of tokens / samples that the expert needs to process in this round, and γ is the measured average computational latency constant; the parameters are obtained by least squares or robust regression fitting. This assumption reflects the high throughput characteristics of GPUs in batch matrix multiplication operations, which can effectively simplify model complexity and maintain the stability of latency prediction.

[0038] Step S2.3: Transmission Overhead Modeling. After measuring the bandwidth B, establish the weight transfer delay and activation round-trip delay, with the following formulas:

[0039]

[0040] Where d is the hidden dimension, This indicates a delay in weight transfer. Indicates activation round-trip delay. Indicates the weight of the experts. Indicates the size of the data type.

[0041] Step S2.4: Combine delay prediction. Combine the calculated delay with the transmission delay.

[0042] "Weighted GPU" path: ; Path to "Activate CPU": ; "GPU is resident" path: .

[0043] The prediction errors and variances of each path are recorded for online calibration. The final integrated delay prediction model is used to compare the costs of different paths.

[0044] The dynamic weight migration and heterogeneous execution scheduling steps involve determining the execution device for each expert during inference based on the current input domain features and the predicted latency model. If an expert resides on a GPU, their weights are directly used for matrix operations. If the expert's weights are located on the CPU or in the cache, the migration cost is evaluated based on the latency model, comparing the expected time of "weight migration to GPU execution" with "activation transfer to CPU execution" to select the optimal path. An asynchronous weight prefetching mechanism is used to migrate experts who may be reactivated in subsequent token inferences to the GPU in advance, improving cache hit rate. The dynamic weight migration and heterogeneous execution scheduling steps include: Step S3.1: During inference, based on the current input task category and feature vector, the expected set of hot experts is determined by calling the expert profile library. Based on the current input domain label or prompt word template, the corresponding set of hot experts in the domain is retrieved from the expert profile library, and a resident list is selected in conjunction with the GPU memory budget.

[0045] Step S3.2: For experts not on the GPU, calculate the migration cost, if If the conditions are met, the weights are transferred to the GPU; otherwise, the activation tensor is passed to the CPU for execution. For the experts selected by gating in this round, the bucket size 's' of each expert is calculated, and the prediction delays for the three paths are calculated respectively. If the expert is already on-premises, execution will proceed directly on the GPU; otherwise, a comparison will be performed. and The smaller option is selected as the execution path, and the decision is logged.

[0046] Step S3.3: Enable asynchronous migration threads to prefetch the weights of potential experts for the next layer during the computation of the previous layer. For experts predicted to be reactivated in subsequent iterations, submit asynchronous migration tasks (independent I / O / computation flow) to overlap with the current layer computation to hide the transfer latency.

[0047] Step S3.4: When GPU memory reaches its limit, "cold experts" are expelled based on their popularity score and cached in host memory. In other words, when memory usage exceeds the threshold, "cold experts" are expelled based on their recent popularity and long-term profile fusion score, and their weights are cached in host memory or the secondary cache.

[0048] Runtime cache update and popularity adjustment steps: The expert activation frequency is updated based on real-time inference statistics, dynamically adjusting the boundary between hot and cold experts. When memory resources change or load fluctuates, expert migration or cache replacement operations are triggered to maintain optimal overall system latency. The cache update and popularity adjustment use a sliding window method to statistically analyze the recent activation frequency of each expert, using this as a short-term popularity indicator. Simultaneously, the short-term popularity is weighted and fused with the long-term popularity in the offline profile library to obtain a comprehensive popularity score, balancing domain priors and runtime dynamics. For example, the weight of short-term popularity can be set to 0.4, and the weight of long-term popularity can be set to 0.6, using the formula: Overall popularity = 0.4 * Short-term popularity + 0.6 * Long-term popularity A comprehensive popularity score is calculated. When GPU memory is insufficient, the system uses this comprehensive popularity score and employs an LRU (Least Recently Used) or similar strategy to evict the expert with the lowest score, freeing up GPU memory space to ensure that the cache always retains the experts most relevant to the current task. Figure 3As shown, the runtime cache update and hotspot adjustment steps include: During inference, the system continuously calculates the real-time activation frequency of each expert using a sliding window and merges it with the expert heat profiles generated in the offline phase. When the online statistical results deviate significantly from the offline profiles (indicating a drift in the input distribution or task domain), the system automatically calculates the similarity between the current activation distribution and the domain profiles in each prefill phase, reselects the best-matching domain profile, and adjusts the long-term prior weights accordingly to enhance the system's adaptability to new tasks. The new fused heat distribution drives the replacement of expert caches between the GPU and CPU, allowing hot experts to reside preferentially on the GPU, thereby maintaining optimal inference latency while keeping GPU memory under control.

[0049] Step S4.1: Real-time statistics of expert activation frequency, and calculation of short-term popularity through a sliding window; Step S4.2: Merge short-term popularity with offline profiles and dynamically adjust the "hot expert residency set"; use a sliding window to count the short-term activation frequency of experts, perform weighted fusion with offline popularity, and dynamically fine-tune the residency list and pre-fetch candidate set.

[0050] Step S4.3: If system load changes or a delay timeout occurs, the degradation execution path is automatically triggered to ensure stability. LRFU / weighted LRU replacement is performed based on a comprehensive metric of "popularity score + access frequency + last usage time," balancing locality and long-term value.

[0051] It is understood that the method can be implemented in a single-machine multi-GPU environment or a CPU+GPU heterogeneous system, and is suitable for hybrid expert model inference tasks under the transformer architecture.

[0052] In this embodiment, when the system receives a reasoning request, it first analyzes the prompts or task descriptions in the request to identify its task domain, such as whether it is a mathematical calculation task, a code generation task, or a common-sense reasoning task. Based on the identified task domain, a group of "hot" experts are pre-fetched from the expert profile library and cached in the GPU cache. These hot experts are those with a high historical activation frequency in the task domain. Pre-loading their weights into the GPU cache can reduce the weight migration time during subsequent reasoning. After reasoning begins, the system processes tokens layer by layer, and the gating network selects appropriate experts based on the input token information. For each selected expert, the dynamic scheduling logic described above is executed to determine whether the expert is in the GPU cache and select the optimal execution path based on the latency model. If the decision is to migrate weights, the expert weights are copied from CPU memory to GPU memory through an asynchronous memory copy operation. This allows overlapping I / O operation time while the GPU is performing calculations, further improving the overall efficiency of the system. During the reasoning process, the system updates the statistical data of expert activation in real time to facilitate subsequent cache updates and hotness adjustments.

[0053] Compared to purely online scheduling, this embodiment can preheat the GPU cache based on the task domain, reducing cold start latency. Through a cost model, namely the comprehensive latency prediction model described above, unnecessary weight migration is avoided, reducing PCIe bus load and end-to-end inference time. This improves GPU memory utilization, keeping only experts most relevant to the current task resident, thus supporting inference of larger models with limited GPU memory.

[0054] Example 2 The present invention also provides a heterogeneous reasoning acceleration system based on hybrid expert models. The heterogeneous reasoning acceleration system based on hybrid expert models can be implemented by executing the process steps of the heterogeneous reasoning acceleration method based on hybrid expert models. That is, those skilled in the art can understand the heterogeneous reasoning acceleration method based on hybrid expert models as a preferred embodiment of the heterogeneous reasoning acceleration system based on hybrid expert models.

[0055] The present invention provides a heterogeneous inference acceleration system based on a hybrid expert model, comprising: performing inference statistics on a multi-domain task dataset; collecting the gating activation frequency of experts at each layer to form a domain-expert heat matrix to reflect the activation preferences of experts under each domain input; determining the set of hot experts that can be accommodated by a second computing unit (GPU) based on offline profiling results; preloading high-frequency activated experts into the storage area of ​​the second computing unit and residing low-frequency activated experts in the memory area of ​​the first computing unit (CPU) to achieve data-driven initial heterogeneous mapping; continuously monitoring the expert activation distribution and gating probability during model inference, using a sliding window mechanism to statistically analyze the dynamic heat distribution of experts, and calculating the degree of deviation from the offline profiling; when activation distribution drift or performance degradation is detected, calculating the migration benefit based on the constant latency difference model (Δ = latency_CPU - latency_GPU) and weight migration cost, and performing limited expert weight migration at stage boundaries (such as the end of prefill or multi-round inference intervals) to achieve adaptive adjustment of expert execution paths.

[0056] Furthermore, the framework and module functional interactions of the heterogeneous reasoning acceleration system based on the hybrid expert model are described in detail below: Offline expert characteristic analysis module: Before model inference, based on representative datasets covering different task contexts, the historical activation behavior of each expert in the hybrid expert model is analyzed; by statistically processing the behavioral data, expert profile data that can characterize the relationship between experts and task contexts is established. In the offline expert characteristic analysis module, the representative dataset includes multiple domains in natural language processing tasks such as mathematical calculation, code generation, and commonsense reasoning; by running the hybrid expert model for inference, and using an analyzer to record the frequency and probability of activation of each expert at each layer, a domain-expert heat matrix is ​​constructed as an expert profile library.

[0057] The execution cost prediction module establishes a cost prediction model for the first and second computing units in the heterogeneous computing environment. This model predicts the cost of executing a computational task in the first computing unit and the cost of transferring expert weights from the first to the second computing unit for execution, based on runtime input features during model inference. The cost in the execution cost prediction model includes time latency; the runtime input features include the amount of data to be processed, the data type, and the task complexity; the first computing unit is a CPU, and the second computing unit is a single GPU.

[0058] Dynamic scheduling execution module: During model inference, when an expert is activated by the gating network, based on pre-established expert profile data and real-time running status, and using the cost prediction model, the system dynamically selects an optimal execution path from multiple candidate execution paths; the candidate execution paths include at least: execution on the first computing unit, execution after migrating the expert weights to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing tasks and data migration operations according to the selected optimal path.

[0059] The online adaptive adjustment module continuously monitors and statistically analyzes real-time expert activation information and / or system status during model inference. Based on the monitored real-time feedback, it dynamically updates expert profile data and / or adjusts the expert residency strategy in the heterogeneous computing environment. In this module, a sliding window is used to statistically analyze the recent activation frequency of each expert as short-term popularity, which is then weighted and fused with the long-term popularity in the offline profile database to obtain a comprehensive popularity score. When the second computing unit's memory is insufficient, an LRU or similar strategy is used to evict the expert with the lowest score based on the comprehensive popularity score.

[0060] Each module of the system of this invention can be deployed independently or executed as a software program embedded in the inference engine. The overall framework enables efficient heterogeneous inference of hybrid expert models in edge or general server environments without altering the model structure.

[0061] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0062] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A heterogeneous inference acceleration method of hybrid expert model, applied to a heterogeneous computing environment comprising a first computing unit and a second computing unit, characterized in that, include: Offline expert characteristic analysis steps: Before model inference, based on a representative dataset covering different task contexts, analyze the historical activation behavior of each expert in the hybrid expert model; by statistically analyzing and processing the data of the behavior, establish expert profile data that can characterize the relationship between experts and task context; The cost prediction step involves establishing a cost prediction model for the first and second computing units in the heterogeneous computing environment. The cost prediction model is used to predict the cost of performing a computing task in the first computing unit and the cost of transferring expert weights from the first computing unit to the second computing unit and having the second computing unit perform the computing task, based on the runtime input features during model inference. Dynamic scheduling execution steps: During the model inference process, when an expert is activated by the gating network, based on the pre-established expert profile data and real-time running status, and using the cost prediction model, an optimal execution path is dynamically selected from multiple candidate execution paths; The candidate execution paths include at least: execution on the first computing unit, execution after migrating the expert weights to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing tasks and data migration operations according to the selected optimal path; Online adaptive adjustment steps: During model inference, continuously monitor and statistically analyze real-time expert activation information and / or system status; dynamically update expert profile data and / or adjust expert residency strategies in heterogeneous computing environments based on the monitored real-time feedback; In the execution cost prediction step, the cost includes time delay; The runtime input characteristics include the amount of data to be processed, the data type, and the complexity of the task. The first computing unit is a CPU, and the second computing unit is a single GPU; For the CPU computing unit, a computational latency model is established using linear fitting, as shown in the following formula: In the formula, α and β are fitting parameters, and s is the number of tokens that the expert needs to process; For GPU computing units, a constant-fit model is used to establish the computational latency, as shown in the following formula: In the formula, γ is a constant; Based on the measured PCIe bandwidth B between the CPU and GPU, the weight transfer latency and activation round-trip latency are established, with the following formulas: where d is the hidden dimension, denotes the weight movement latency, denotes the activation round trip latency, denotes the expert weight size, denotes the data type size; In the online adaptive adjustment step, the recent activation frequency of each expert is statistically analyzed using a sliding window as the short-term popularity, and then weighted and integrated with the long-term popularity in the offline profile database to obtain a comprehensive popularity score. When the second computing unit's video memory is insufficient, the LRU strategy is adopted based on the comprehensive popularity score to evict the expert with the lowest score.

2. The heterogeneous reasoning acceleration method for hybrid expert models according to claim 1, characterized in that, In the offline expert feature analysis step, the representative dataset includes multiple domains in natural language processing tasks, such as mathematical calculation, code generation, and common sense reasoning. By running a hybrid expert model for inference and using an analyzer to record the frequency and probability of activation of each expert at each layer, a domain-expert heat matrix is ​​constructed as an expert profile library.

3. The heterogeneous reasoning acceleration method for hybrid expert models according to claim 1, characterized in that, In the dynamic scheduling execution step, when an expert is activated, if the expert weight has already been stored in the second computing unit cache, it will be executed directly in the second computing unit. If the expert weights are in the memory of the first computing unit, then the cost prediction model is used to compare the cost of migrating the weights to the second computing unit for execution. The cost of activating the transfer to the first computing unit for execution. Choose the path with lower cost to execute, as shown in the following formula: In the formula, This indicates a delay in weight transfer. This indicates the activation round-trip delay.

4. The heterogeneous reasoning acceleration method for hybrid expert models according to claim 1, characterized in that, When the online statistical results deviate too much from the offline profiles, the similarity between the current activation distribution and the profiles of each domain is automatically calculated in each pre-filling stage. The best matching domain profile is then selected, and the long-term prior weights are adjusted accordingly.

5. A heterogeneous reasoning acceleration system based on hybrid expert models, characterized in that, To implement the method according to any one of claims 1 to 4, comprising: Offline expert characteristic analysis module: Before model inference, based on a representative dataset covering different task contexts, the historical activation behavior of each expert in the hybrid expert model is analyzed; by statistically processing the data of the behavior, expert profile data that can characterize the relationship between experts and task context is established; Execution cost prediction module: For the first computing unit and the second computing unit in the heterogeneous computing environment, a cost prediction model is established; the cost prediction model is used to predict the cost of executing a computing task in the first computing unit and the cost of transferring expert weights from the first computing unit to the second computing unit and having the second computing unit execute the computing task, based on the runtime input characteristics during model inference. Dynamic scheduling execution module: During model inference, when an expert is activated by the gating network, based on pre-established expert profile data and real-time running status, and using the cost prediction model, the system dynamically selects an optimal execution path from multiple candidate execution paths; the candidate execution paths include at least: execution on the first computing unit, execution after migrating the expert weights to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing tasks and data migration operations according to the selected optimal path; Online adaptive adjustment module: During model inference, continuously monitor and statistically analyze real-time expert activation information and / or system status; dynamically update expert profile data and / or adjust expert residency strategies in heterogeneous computing environments based on the monitored real-time feedback.

6. The heterogeneous inference acceleration system based on a hybrid expert model according to claim 5, characterized in that, In the offline expert feature analysis module, the representative dataset includes multiple domains of natural language processing tasks, such as mathematical calculation, code generation, and common sense reasoning. By running a hybrid expert model for inference and using an analyzer to record the frequency and probability of activation of each expert at each layer, a domain-expert heat matrix is ​​constructed as an expert profile library. The execution cost prediction module includes time delay as a cost; the runtime input features include the amount of data to be processed, the data type, and the task complexity; the first computing unit is a CPU, and the second computing unit is a single GPU.

7. The heterogeneous inference acceleration system based on a hybrid expert model according to claim 5, characterized in that, In the online adaptive adjustment module, a sliding window is used to count the recent activation frequency of each expert as short-term popularity, and this is weighted and fused with the long-term popularity in the offline profile database to obtain a comprehensive popularity score. When the second computing unit's video memory is insufficient, the LRU strategy is adopted based on the comprehensive popularity score to evict the expert with the lowest score.

Images