Heterogeneous reasoning acceleration method and system for hybrid expert model

Abstract

The invention provides a heterogeneous reasoning acceleration method and system for a hybrid expert model, and the method comprises the steps: analyzing the historical activation behavior of each expert based on a representative data set of a multi-task context, constructing an expert portrait representing the relevance between the experts and tasks, and providing priori knowledge for scheduling; a cost prediction model is established for the heterogeneous computing units, and time delays of different execution paths are predicted according to input characteristics (such as data volume) during operation; in the reasoning process, an expert portrait and a cost prediction model are combined, an optimal path is selected from candidate paths (CPU execution, GPU direct execution and weight migration to GPU execution) for an activated expert, and task execution and data migration are controlled; based on real-time expert activation statistics and a system state, an expert portrait and a resident strategy of an expert in a heterogeneous environment are dynamically updated, and a feedback closed loop is formed. According to the method, adaptive optimization of the expert reasoning path is realized, and the problem of performance degradation of a traditional static distribution strategy in a complex input scene is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence reasoning optimization, in particular to a heterogeneous reasoning acceleration method and system for mixed expert models. Especially, it relates to a heterogeneous reasoning acceleration method and system for mixed expert models based on input feature perception and dynamic weight migration. BACKGROUND

[0002] In recent years, large-scale language models represented by the Mixture-of-Experts (MoE) architecture have been widely applied. This architecture uses a sparse activation mechanism to call only a small number of expert subnetworks when processing each input, effectively controlling the amount of computation while expanding the model capacity. However, in actual deployment, especially in resource-constrained heterogeneous computing environments such as GPU memory, the inference efficiency of MoE models faces significant bottlenecks.

[0003] Existing systems usually use fixed expert division or simple hot expert resident mechanisms, which fail to dynamically adjust according to input task domains or semantic features, resulting in GPU resource waste or loading delays during cross-domain inference. In resource-constrained environments (such as single machines or end-side), GPU memory is not enough to accommodate all expert weights at the same time, and frequent data migration between CPU and GPU leads to increased transmission delays. Traditional scheduling strategies rely on fixed thresholds or heuristic judgments and do not consider the impact of dynamic factors such as input token number, sparsity, and batch size on computation delay.

[0004] Therefore, there is a need for an efficient MoE inference method that can adaptively allocate experts between heterogeneous devices, dynamically predict delays based on inputs, and migrate weights in real time. SUMMARY

[0005] To overcome the defects in the prior art, the present application aims to provide a heterogeneous reasoning acceleration method and system for mixed expert models in a resource-constrained environment, which can solve the memory bottleneck and delay fluctuation problems of the Mixture-of-Experts (MoE) architecture during inference. It is applied to a heterogeneous computing environment containing a first computing unit (such as a CPU) and a second computing unit (such as a GPU).

[0006] According to the heterogeneous reasoning acceleration method for mixed expert models provided by the present application, it is applied to a heterogeneous computing environment containing a first computing unit and a second computing unit, which includes: Offline expert characteristic analysis step: Before model inference, based on representative datasets covering different task contexts, analyze the historical activation behavior of each expert in the mixed expert model; by statistically processing the data of the behavior, establish expert portrait data that can represent the association between experts and task contexts; The execution cost prediction step: a cost prediction model is established for the first computing unit and the second computing unit in the heterogeneous computing environment; the cost prediction model is used to predict the cost of executing a computing task on the first computing unit and the cost of migrating the expert weight from the first computing unit to the second computing unit and executing the computing task by the second computing unit according to the runtime input features during model inference; The dynamic scheduling execution step: when an expert is activated by the gating network during model inference, an optimal execution path is dynamically selected from multiple candidate execution paths based on the pre-established expert portrait data and real-time running state and by using the cost prediction model; the candidate execution paths at least include: execution on the first computing unit, execution after migrating the expert weight to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing task and data migration operation according to the selected optimal path; The online adaptive adjustment step: during model inference, real-time expert activation information and / or system state are continuously monitored and counted; the expert portrait data are dynamically updated and / or the expert residence strategy in the heterogeneous computing environment is adjusted according to the monitored real-time feedback. Preferably, in the offline expert characteristic analysis step, the representative data set includes multiple fields such as mathematical calculation, code generation and common sense reasoning in natural language processing tasks; The frequency and probability of activation of each expert in each layer are recorded by running the hybrid expert model for inference and using the analyzer, and a field-expert heat matrix is constructed as an expert portrait library. Preferably, in the execution cost prediction step, the cost includes time delay; The runtime input features include the amount of data to be processed, the data type and the task complexity; The first computing unit is a CPU, and the second computing unit is a single GPU.

[0007] Preferably, for the CPU computing unit, a linear fitting is used to establish a computing delay model, and the formula is as follows:

[0008] In the formula, α and β are fitting parameters, and s is the number of tokens that need to be processed by the expert; For the GPU computing unit, a constant fitting is used to establish a computing delay model, and the formula is as follows:

[0009] In the formula, γ is a constant; According to the measured bandwidth B between the CPU and the GPU, the weight transfer delay and the activation round-trip delay are established, and are respectively as follows:

[0010]

[0011] wherein d is a hidden dimension, represents the weight transfer delay, represents the activation round-trip delay, represents the expert weight size, represents the data type size. Preferably, in the dynamic scheduling execution step, when an expert is activated, if the expert weight has been resident in the cache of the second computing unit, then the execution is directly performed in the second computing unit; if the expert weight is in the memory of the first computing unit, then the cost prediction model is used to compare the cost of weight migration to the second computing unit for execution and the cost of activation transmission to the first computing unit for execution , and the path with smaller cost is selected for execution, and the formula is as follows:

[0012]

[0013] wherein, represents the weight transfer delay, represents the activation round-trip delay.

[0014] Preferably, in the online adaptive adjustment step, the recent activation frequency of each expert is counted by using a sliding window as a short-term heat, and the long-term heat in the offline portrait library is weighted and fused to obtain a comprehensive heat score; When the second computing unit memory is insufficient, according to the comprehensive heat score, an LRU or similar strategy is used to evict the expert with the lowest score.

[0015] Preferably, when the online statistical result deviates too much from the offline portrait, the similarity between the current activation distribution and each domain portrait is automatically calculated at each pre-filling stage, the most matched domain portrait is reselected, and the long-term prior weight is adjusted accordingly.

[0016] According to the heterogeneous inference acceleration system of the mixed expert model provided by the application, comprising: An offline expert characteristic analysis module: before model inference, based on representative data sets covering different task contexts, the historical activation behaviors of each expert in the mixed expert model are analyzed; by counting and processing the data of the behaviors, expert portrait data capable of representing the correlation between the expert and the task context are established; The execution cost prediction module: for a first computing unit and a second computing unit in the heterogeneous computing environment, a cost prediction model is established; the cost prediction model is used to predict the cost of executing a computing task at the first computing unit and the cost of migrating the expert weight from the first computing unit to the second computing unit and executing the computing task by the second computing unit according to the runtime input features during model inference; The dynamic scheduling execution module: during the model inference process, when an expert is activated by the gating network, based on the pre-established expert portrait data and the real-time running state, and using the cost prediction model, an optimal execution path is dynamically selected from multiple candidate execution paths; the candidate execution paths at least include: execution on the first computing unit, execution after migrating the expert weight to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing task and data migration operation according to the selected optimal path; The online adaptive adjustment module: during the model inference process, the real-time expert activation information and / or system state are continuously monitored and counted; according to the monitored real-time feedback, the expert portrait data is dynamically updated, and / or the expert residence strategy in the heterogeneous computing environment is adjusted.

[0017] Preferably, in the offline expert characteristic analysis module, the representative data set includes mathematical calculation, code generation, common sense reasoning in natural language processing tasks, and multiple fields; By running the hybrid expert model for inference, and using the analyzer to record the frequency and probability of each expert being activated in each layer, a field-expert heat matrix is constructed as an expert portrait library; In the execution cost prediction model, the cost includes time delay; the runtime input features include the amount of data to be processed, the data type, and the task complexity; the first computing unit is a CPU, and the second computing unit is a single GPU.

[0018] Preferably, in the online adaptive adjustment module, the recent activation frequency of each expert is counted as a short-term heat by using a sliding window, and is weighted and fused with the long-term heat in the offline portrait library to obtain a comprehensive heat score; When the second computing unit is short of memory, according to the comprehensive heat score, an LRU or similar strategy is used to evict the expert with the lowest score.

[0019] Compared with the prior art, the present application has the following beneficial effects: 1. The application solves the technical problems of unbalanced use of experts in cross-domain scenarios and difficulty in effectively reusing GPU high-speed storage by expert activation statistics and clustering analysis of samples in different task domains (such as mathematics, code, translation, common sense reasoning, etc.) to obtain a "domain-expert heat matrix".

[0020] 2. The application fits the CPU / GPU calculation delay curve according to input features such as token quantity, activation sparsity, batch size, and fuses the weight carrying time of PCIe / NVLink transmission bandwidth, solves the problem that the existing heuristic threshold strategy cannot quantify the real cost of "transmit weight vs. transmit activation", and makes the device selection predictable and interpretable.

[0021] 3. The application compares the comprehensive delay of "weight migration to GPU execution" and "activation transmission to CPU execution" at runtime, and adaptively determines the execution path and resident device of the expert by combining offline profiling and online statistics, solving the problem of frequent loading jitter caused by low efficiency of CPU / GPU cooperation and insufficient video memory.

[0022] 4. The application prefetches the weights of experts that may be activated during the next layer / iteration calculation, and fuses the heat with a sliding window combined with long-term profiling, reducing the cold start overhead and improving the expert cache hit rate, thereby reducing the average response time and tail latency.

[0023] 5. The application significantly reduces GPU memory occupancy (only hot experts are long-term resident), reduces PCIe transmission volume, improves end-to-end throughput and stability through a dual-channel strategy of "domain priori + runtime feedback", and does not depend on model structure modification, and is universal and portable for multiple MoE architectures and various hardware forms. BRIEF DESCRIPTION OF DRAWINGS

[0024] Other features, objects and advantages of the application will become more apparent through reading the following detailed description of the non-limiting embodiments with reference to the accompanying drawings: Figure 1 The figure is a schematic diagram of the overall system architecture of the application.

[0025] Figure 2 The figure is a schematic diagram of offline expert profiling in the embodiment of the application.

[0026] Figure 3 The figure is a schematic diagram of the runtime expert migration and cache scheduling process in the embodiment of the application. DETAILED DESCRIPTION

[0027] The application will be described in detail below with specific embodiments. The following examples will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of protection of the application.

[0028] Example 1 This embodiment shows a reasoning acceleration method for a hybrid expert model in a heterogeneous computing environment containing a CPU (first computing unit) and a single GPU (second computing unit). The method establishes a heat map by offline analyzing the activation frequency of experts in different task domains (such as mathematics, code), establishes a linear / constant delay model based on the number of input tokens online, and combines a bandwidth model to predict the total delay. Finally, it dynamically decides whether the expert is executed on the CPU or migrated to the GPU for execution, while updating the expert heat through a sliding window to adjust the GPU cache.

[0029] According to the hybrid expert model reasoning acceleration method provided by the application, as shown in Figure 1 , the method comprises the following steps: An offline expert portrait generation step: by collecting data sets of different task domains, recording the activation feature information of each expert in the reasoning process, and constructing a domain-expert heat matrix. The activation features include not only the frequency of expert selection, but also the average gating probability, computation load or output energy, etc. to reflect the comprehensive activity of the expert in the field. Cluster or dimensionality reduction algorithm is used to analyze the activation feature information to obtain the activity weight of each expert in different fields, and generate an offline expert portrait library as a priori basis for runtime expert allocation. The data set of different task domains includes representative task domain data sets such as mathematics, code, translation, common sense reasoning, etc. As shown in Figure 2 , the offline expert portrait generation step comprises: Step S1.1: Collect data sets of different task domains and preprocess the data of the data sets. The preprocessing includes unifying the text length distribution, tokenizer and decoding strategy, eliminating abnormal samples and repeated samples, and ensuring consistent statistical scope.

[0030] Step S1.2: Perform reasoning on the hybrid expert, and record the activation count and probability distribution of each expert in each layer. Without changing the model structure and weight, perform reasoning on each sample, record the expert selection result and its probability (top-k index and weight) of the gate output, and aggregate the layer expert activation vector by token / sample.

[0031] Step S1.3: Perform model inference on each task domain dataset, and count the number of times each layer expert is selected by the gating module in this domain. In each forward calculation, the gating layer outputs the score vector of all experts, and the system records the number of the top k expert and accumulates the number of its activations.

[0032] For all samples in domain d, record the total number of times expert e is activated as , and the total number of tokens processed as . Then the average activation frequency of expert e in domain d is defined as:

[0033] Arrange the values of all layers and all experts to form a "domain-expert activation frequency matrix" to represent the usage heat of each expert in different domains.

[0034] Optionally, perform principal component analysis (PCA) or clustering operations on the matrix data to identify groups of experts with similar activation distributions, thereby forming a hierarchical expert heat table to provide a basis for subsequent expert placement and heterogeneous mapping.

[0035] Step S1.4: Identify expert group structure through inter-layer clustering algorithms such as K-means, DBSCAN, or PCA dimensionality reduction to form an expert portrait library; bind the expert heat table with the model weight index to support subsequent runtime loading optimization. Bind the heat matrix with the storage index of the expert weight to form an expert portrait library to provide a prior list for runtime hot expert residency and candidate pre-fetching.

[0036] Comprehensive delay prediction step: During model runtime, collect the actual execution time of experts on CPU and GPU according to input sequence length, token activation number, batch size, task type, and other input features, and fit to obtain the functional relationship between input size and delay; measure the weight transmission delay curve based on the data bandwidth of PCIe or NVLink, and combine the calculation delay curve to build a comprehensive delay prediction model for subsequent dynamic scheduling decisions. The comprehensive delay prediction step includes: Step S2.1: Feature definition and sampling. Define input size features, including sequence length, the number of activated tokens s in the current iteration, batch size, beam width, sparsity index, etc.; sample and run several rounds according to different values.

[0037] Step S2.2: CPU / GPU calculation delay fitting, the formulas are as follows: The CPU delay model uses linear fitting: ; The GPU latency model adopts a constant fitting form: ; where s is the number of tokens / samples to be processed by the expert in this round, and γ is the measured average computation latency constant; the parameters are fitted by least squares or robust regression. This assumption reflects the high throughput characteristics of the GPU in batch matrix multiplication operations, which can effectively simplify the model complexity and maintain the stability of the latency prediction.

[0038] Step S2.3: Transmission overhead modeling. After measuring the bandwidth B, the weight transfer delay and the activation round-trip delay are established, and the formulas are as follows:

[0039]

[0040] where d is the hidden dimension, represents the weight transfer delay, represents the activation round-trip delay, represents the expert weight size, represents the data type size.

[0041] Step S2.4: Integrated latency prediction. The computation delay and the transmission delay are combined.

[0042] “Weight on GPU” path: ;“Activation on CPU” path: ; “GPU resident” path: . Record the prediction error and variance of each path for online calibration. The final integrated delay prediction model is used to compare the costs of different paths.

[0043] Dynamic weight migration and heterogeneous execution scheduling step; during inference, the execution device of each expert is determined according to the domain features of the current input and the predicted latency model; if the expert is resident on the GPU, its weight is directly called for matrix operation; if the expert weight is located on the CPU or the cache area, the migration cost is evaluated according to the latency model, the expected time of “weight migration to GPU execution” and “activation transmission to CPU execution” is compared, and the optimal path is selected; through the asynchronous weight prefetch mechanism, the experts that may be activated again in subsequent token inference are migrated into the GPU in advance, improving the cache hit rate. The dynamic weight migration and heterogeneous execution scheduling step includes:

[0044] Dynamic weight migration and heterogeneous execution scheduling step; during inference, the execution device of each expert is determined according to the domain features of the current input and the predicted latency model; if the expert is resident on the GPU, its weight is directly called for matrix operation; if the expert weight is located on the CPU or the cache area, the migration cost is evaluated according to the latency model, the expected time of “weight migration to GPU execution” and “activation transmission to CPU execution” is compared, and the optimal path is selected; through the asynchronous weight prefetch mechanism, the experts that may be activated again in subsequent token inference are migrated into the GPU in advance, improving the cache hit rate. The dynamic weight migration and heterogeneous execution scheduling step includes: Step S3.1: During inference, based on the current input task category and feature vector, the expected set of hot experts is determined by calling the expert profile library. Based on the current input domain label or prompt word template, the corresponding set of hot experts in the domain is retrieved from the expert profile library, and a resident list is selected in conjunction with the GPU memory budget.

[0045] Step S3.2: For experts not on the GPU, calculate the migration cost, if If the conditions are met, the weights are transferred to the GPU; otherwise, the activation tensor is passed to the CPU for execution. For the experts selected by gating in this round, the bucket size 's' of each expert is calculated, and the prediction delays for the three paths are calculated respectively. If the expert is already on-premises, execution will proceed directly on the GPU; otherwise, a comparison will be performed. and The smaller option is selected as the execution path, and the decision log is recorded.

[0046] Step S3.3: Enable asynchronous migration threads to prefetch the weights of potential experts for the next layer during the computation of the previous layer. For experts predicted to be reactivated in subsequent iterations, submit asynchronous migration tasks (independent I / O / computation flow) to overlap with the current layer computation to hide the transfer latency.

[0047] Step S3.4: When GPU memory reaches its limit, "cold experts" are expelled based on their popularity score and cached in host memory. In other words, when memory usage exceeds the threshold, "cold experts" are expelled based on their recent popularity and long-term profile fusion score, and their weights are cached in host memory or the secondary cache.

[0048] Runtime cache update and popularity adjustment steps: The expert activation frequency is updated based on real-time inference statistics, dynamically adjusting the boundary between hot and cold experts. When memory resources change or load fluctuates, expert migration or cache replacement operations are triggered to maintain optimal overall system latency. The cache update and popularity adjustment use a sliding window method to statistically analyze the recent activation frequency of each expert, using this as a short-term popularity indicator. Simultaneously, the short-term popularity is weighted and fused with the long-term popularity in the offline profile library to obtain a comprehensive popularity score, balancing domain priors and runtime dynamics. For example, the weight of short-term popularity can be set to 0.4, and the weight of long-term popularity can be set to 0.6, using the formula: Overall popularity = 0.4 * Short-term popularity + 0.6 * Long-term popularity A comprehensive popularity score is calculated. When GPU memory is insufficient, the system uses this comprehensive popularity score and employs an LRU (Least Recently Used) or similar strategy to evict the expert with the lowest score, freeing up GPU memory space to ensure that the cache always retains the experts most relevant to the current task. Figure 3As shown, the runtime cache updating and hotness adjusting step includes: In the inference process, the system continuously calculates the real-time activation frequency of each expert based on a sliding window, and fuses it with the expert hotness portrait generated in the offline stage. When the online statistical result deviates too much from the offline portrait (indicating that the input distribution or task domain has drifted), the system automatically calculates the similarity between the current activation distribution and the portrait of each domain in each prefill stage, reselects the most matched domain portrait, and adjusts the long-term prior weight accordingly to enhance the adaptability of the system to new tasks. The new fused hotness distribution will drive the expert cache replacement between GPU and CPU, so that the hot experts are preferentially resident in GPU, thereby maintaining the optimal state of inference delay while keeping the video memory controllable.

[0049] Step S4.1: Real-time statistics of expert activation frequency, short-term hotness is calculated through a sliding window; Step S4.2: Fuse short-term hotness with offline portrait, dynamically adjust the "hot expert resident set"; use a sliding window to calculate the short-term activation frequency of the experts, and fuse it with the offline hotness for weighted fusion, to dynamically fine-tune the resident list and prefetch candidate set.

[0050] Step S4.3: If the system load changes or delay timeout occurs, automatically trigger the degradation execution path to ensure stability. According to the comprehensive index of "hotness score + access frequency + last use time", LRFU / weighted LRU replacement is performed, taking into account locality and long-term value.

[0051] It can be understood that the method can be implemented on a single machine multi-GPU environment or a CPU+GPU heterogeneous system, and is suitable for mixed expert model inference tasks under the transformer architecture.

[0052] In this embodiment, when the system receives an inference request, it first analyzes the prompt or task description in the request to identify the task domain, such as determining whether it is a mathematical calculation task, a code generation task, or a common sense reasoning task, etc. According to the identified task domain, a group of hot experts in the expert portrait library is pre-fetched to the GPU cache, these hot experts are experts with high historical activation frequency in the task domain, pre-loading their weights into the GPU cache can reduce the weight migration time in the subsequent inference process. After the inference starts, the system processes the tokens layer by layer, and the gating network selects the appropriate experts according to the input token information. For each selected expert, the above dynamic scheduling logic is executed to determine whether the expert is in the GPU cache, and the optimal execution path is selected according to the delay model. If the decision is to migrate the weight, the expert weight is copied from the CPU memory to the GPU video memory through the asynchronous memory copy operation, so that the I / O operation time can be overlapped with the GPU calculation, further improving the overall efficiency of the system. During the inference process, the system updates the statistical data of expert activation in real time to facilitate subsequent cache updating and hotness adjustment.

[0053] Compared with pure online scheduling, the embodiment can preheat the GPU cache based on the task domain to reduce the cold start delay. Through the cost model, i.e. the above-mentioned comprehensive delay prediction model, unnecessary weight migration is avoided, the PCIe bus load and end-to-end inference time are reduced. The GPU video memory utilization is improved, only the experts most relevant to the current task are resident, thereby supporting the inference of larger models under limited video memory.

[0054] Embodiment 2 The application also provides a heterogeneous inference acceleration system of a mixed expert model, which can be realized by executing the flow steps of the heterogeneous inference acceleration method of the mixed expert model, that is, the heterogeneous inference acceleration method of the mixed expert model can be understood by those skilled in the art as the preferred implementation of the heterogeneous inference acceleration system of the mixed expert model.

[0055] The present invention provides a heterogeneous inference acceleration system based on a hybrid expert model, comprising: performing inference statistics on a multi-domain task dataset; collecting the gating activation frequency of experts at each layer to form a domain-expert heat matrix to reflect the activation preferences of experts under each domain input; determining the set of hot experts that can be accommodated by a second computing unit (GPU) based on offline profiling results; preloading high-frequency activated experts into the storage area of ​​the second computing unit and residing low-frequency activated experts in the memory area of ​​the first computing unit (CPU) to achieve data-driven initial heterogeneous mapping; continuously monitoring the expert activation distribution and gating probability during model inference, using a sliding window mechanism to statistically analyze the dynamic heat distribution of experts, and calculating the degree of deviation from the offline profiling; when activation distribution drift or performance degradation is detected, calculating the migration benefit based on the constant latency difference model (Δ = latency_CPU - latency_GPU) and weight migration cost, and performing limited expert weight migration at stage boundaries (such as the end of prefill or multi-round inference intervals) to achieve adaptive adjustment of expert execution paths.

[0056] Furthermore, the framework and module functional interactions of the heterogeneous reasoning acceleration system based on the hybrid expert model are described in detail below: Offline expert characteristic analysis module: Before model inference, based on representative datasets covering different task contexts, the historical activation behavior of each expert in the hybrid expert model is analyzed; by statistically processing the behavioral data, expert profile data that can characterize the relationship between experts and task contexts is established. In the offline expert characteristic analysis module, the representative dataset includes multiple domains in natural language processing tasks such as mathematical calculation, code generation, and commonsense reasoning; by running the hybrid expert model for inference, and using an analyzer to record the frequency and probability of activation of each expert at each layer, a domain-expert heat matrix is ​​constructed as an expert profile library.

[0057] The execution cost prediction module establishes a cost prediction model for the first and second computing units in the heterogeneous computing environment. This model predicts the cost of executing a computational task in the first computing unit and the cost of transferring expert weights from the first to the second computing unit for execution, based on runtime input features during model inference. The cost in the execution cost prediction model includes time latency; the runtime input features include the amount of data to be processed, the data type, and the task complexity; the first computing unit is a CPU, and the second computing unit is a single GPU.

[0058] The dynamic scheduling execution module: in the model inference process, when an expert is activated by the gating network, based on the pre-established expert portrait data and the real-time running state, and using the cost prediction model, an optimal execution path is dynamically selected from multiple candidate execution paths; the candidate execution paths at least include: executing on the first computing unit, executing after migrating the expert weight to the second computing unit, and directly executing on the second computing unit; the system controls the corresponding computing task and data migration operation according to the selected optimal path.

[0059] The online adaptive adjustment module: in the model inference process, the real-time expert activation information and / or system state are continuously monitored and counted; according to the monitored real-time feedback, the expert portrait data is dynamically updated, and / or the expert residence strategy in the heterogeneous computing environment is adjusted. In the online adaptive adjustment module, the recent activation frequency of each expert is counted as a short-term heat using a sliding window, and is weighted and fused with the long-term heat in the offline portrait library to obtain a comprehensive heat score; when the second computing unit runs out of memory, according to the comprehensive heat score, the LRU or similar strategy is used to evict the expert with the lowest score.

[0060] The modules of the system of the present application can be independently deployed or embedded in the inference engine as a software program for execution. The overall framework can realize efficient heterogeneous inference of the mixed expert model in the end-side or general server environment without changing the model structure.

[0061] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module, unit thereof in a pure computer readable program code manner, the same functions can be realized by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the system provided by the present application and each device, module, unit thereof can be considered as a hardware component, and the devices, modules, units included therein for realizing various functions can also be considered as structures within the hardware component; the devices, modules, units for realizing various functions can also be considered as both software modules realizing the method and structures within the hardware component.

[0062] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be arbitrarily combined with each other without conflict.

Claims

1. A heterogeneous inference acceleration method of hybrid expert model, applied to a heterogeneous computing environment comprising a first computing unit and a second computing unit, characterized in that, Comprise: Offline expert characteristic analysis step: before model inference, based on representative dataset covering different task context, analyze the historical activation behavior of each expert in the hybrid expert model; by statistics and processing the data of the behavior, establish expert portrait data that can represent the correlation between the expert and the task context; Execution cost prediction step: for the first computing unit and the second computing unit in the heterogeneous computing environment, establish a cost prediction model; the cost prediction model is used to predict the cost of executing a computing task on the first computing unit, and the cost of migrating the expert weight from the first computing unit to the second computing unit and executing the computing task by the second computing unit according to the runtime input features during model inference; Dynamic scheduling execution step: during model inference, when an expert is activated by the gating network, based on the pre-established expert portrait data and real-time running state, and using the cost prediction model, dynamically select an optimal execution path from multiple candidate execution paths; The candidate execution paths at least include: execution on the first computing unit, execution after migrating the expert weight to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing task and data migration operation according to the selected optimal path; Online adaptive adjustment step: during model inference, continuously monitor and statistics the real-time expert activation information and / or system state; dynamically update the expert portrait data and / or adjust the expert residence strategy in the heterogeneous computing environment according to the monitored real-time feedback.

2. The heterogeneous inference acceleration method of hybrid expert model according to claim 1, wherein, In the offline expert characteristic analysis step, the representative dataset includes mathematical calculation, code generation, common sense reasoning in natural language processing tasks, and multiple fields; By running the hybrid expert model for inference, and using the analyzer to record the frequency and probability of each expert being activated in each layer, a field-expert heat matrix is constructed as an expert portrait library.

3. The heterogeneous inference acceleration method of hybrid expert modeling of claim 1, wherein, In the execution cost prediction step, the cost includes time delay; The runtime input features include the amount of data to be processed, data type, and task complexity; The first computing unit is CPU, and the second computing unit is single GPU.

4. The heterogeneous reasoning acceleration method for hybrid expert models according to claim 3, characterized in that, For CPU computing unit, linear fitting is used to establish the calculation delay model, the formula is as follows: In the formula, α and β are fitting parameters, and s is the number of tokens that the expert needs to process; For GPU computing unit, constant fitting is used to establish the calculation delay model, the formula is as follows: In the formula, γ is a constant; According to the actual bandwidth B between CPU and GPU through PCIe, the weight carrying delay and activation round trip delay are established, and the formulas are as follows: where d is the hidden dimension, denotes the weight movement latency, denotes the activation round trip latency, denotes the expert weight size, denotes the data type size.

5. The heterogeneous inference acceleration method of hybrid expert modeling of claim 1, wherein, In the dynamic scheduling execution step, when an expert is activated, if the expert weight has been resident in the second computing unit cache, it is directly executed on the second computing unit; If the expert weights are in the first compute unit memory, then compare the cost of migrating the weights to the second compute unit execution using the cost prediction model to the cost of activating the transfer to the first compute unit execution and select the path with the lower cost to execute, as follows: In the formula, denotes the weight transport delay, denotes the activation round trip delay.

6. The heterogeneous reasoning acceleration method for hybrid expert models according to claim 1, characterized in that, In the online adaptive adjustment step, the sliding window is used to statistics the recent activation frequency of each expert as short-term heat, and the long-term heat in the offline portrait library is weighted and fused to obtain a comprehensive heat score; When the second computing unit runs out of memory, according to the comprehensive heat score, the LRU or similar strategy is used to evict the expert with the lowest score.

7. The heterogeneous inference acceleration method of hybrid expert modeling of claim 1, wherein, When the online statistical result deviates too much from the offline portrait, the similarity between the current active distribution and the domain portrait is automatically calculated at each pre-filling stage, the most matching domain portrait is reselected, and the long-term prior weight is adjusted accordingly.

8. A heterogeneous inference acceleration system of hybrid expert models, characterized in that, Comprise: An offline expert characteristic analysis module: before model inference, based on a representative dataset covering different task contexts, the historical activation behavior of each expert in the mixed expert model is analyzed; by statistically processing the data of the behavior, an expert portrait data capable of representing the correlation between the expert and the task context is established; An execution cost prediction module: for the first computing unit and the second computing unit in the heterogeneous computing environment, a cost prediction model is established; the cost prediction model is used to predict the cost of executing a computing task on the first computing unit, and the cost of migrating the expert weight from the first computing unit to the second computing unit and executing the computing task by the second computing unit according to the runtime input features during model inference; A dynamic scheduling execution module: during model inference, when an expert is activated by the gating network, based on the pre-established expert portrait data and real-time running state, and using the cost prediction model, an optimal execution path is dynamically selected from multiple candidate execution paths; The candidate execution paths at least include: execution on the first computing unit, execution after migrating the expert weight to the second computing unit, and direct execution on the second computing unit; the system controls the corresponding computing task and data migration operation according to the selected optimal path; An online adaptive adjustment module: during model inference, real-time expert activation information and / or system state are continuously monitored and statistically processed; according to the monitored real-time feedback, the expert portrait data is dynamically updated, and / or the expert's residence strategy in the heterogeneous computing environment is adjusted.

9. The heterogeneous inference acceleration system of hybrid expert models of claim 8, wherein, In the offline expert characteristic analysis module, the representative dataset includes multiple fields such as mathematical calculation, code generation, and common sense reasoning in natural language processing tasks; By running the mixed expert model for inference, and using the analyzer to record the frequency and probability of each expert being activated at each layer, a field-expert heat matrix is constructed as an expert portrait library; In the execution cost prediction model, the cost includes time delay; the runtime input features include the amount of data to be processed, data type, and task complexity; the first computing unit is CPU, and the second computing unit is single GPU.

10. The heterogeneous inference acceleration system of hybrid expert models of claim 8, wherein, In the online adaptive adjustment module, the recent activation frequency of each expert is statistically processed as short-term heat, and is weighted and fused with the long-term heat in the offline portrait library to obtain a comprehensive heat score; When the second computing unit runs out of memory, according to the comprehensive heat score, the LRU or similar strategy is used to evict the expert with the lowest score.

Images