Hybrid expert model asynchronous parallel inference method, system, device and medium

Abstract

The application discloses a kind of mixed expert model asynchronous parallel inference method, system, equipment and medium, they are corresponding scheme, in scheme: the synchronization of the inter-gpu calculation and communication caused by full-to-full set communication in expert parallel is decoupled, allow model calculation and token data communication asynchronous parallel, to sufficiently cover the data communication overhead caused by expert parallel, and eliminate synchronization waiting overhead;For the uneven phenomenon of expert in inference, hot expert is placed in GPU first, and cold expert is side-loaded in CPU, to release GPU video memory space, and the calculation efficiency of GPU can be improved by increasing batch size during inference;By dynamically selecting the most suitable execution unit and the cold expert that needs to be loaded, efficient resource scheduling is achieved;Overall, by the present application, the communication overhead and waiting overhead during expert parallel inference can be significantly reduced, while the calculation efficiency of GPU is improved, and the overall throughput performance during inference is optimized.

Description

Technical Field

[0001] This invention relates to the field of hybrid expert model reasoning, and more particularly to an asynchronous parallel reasoning method, system, device, and medium for hybrid expert models. Background Technology

[0002] In recent years, deep neural network architectures, represented by the Transformer model, have brought about profound changes in the field of natural language processing. The core innovation of the Transformer lies in its attention mechanism, which achieves efficient parallel matrix computation and the ability to handle long-distance dependencies by linearly modeling the dependencies between different positions in the input sequence. This architecture has greatly promoted the development of large-scale pre-trained models and has become the foundation for many natural language generation and understanding models.

[0003] As the scale of large language models continues to expand, traditional dense architectures, while significantly increasing the number of parameters and expressive power, also incur enormous computational overhead. This is because for each token in the input, all parameters must be activated in the linearly extended feed-forward network (FFN) layer of each layer of the model for processing. To alleviate the computational resource demands during the training and inference of large language models, academia and industry have begun to actively explore sparse structures in recent years, with the Mixture of Experts (MoE) architecture becoming a typical example. In the MoE architecture, the FFN layers are replaced by MoE layers, which consist of multiple small linear matrices called experts and activation functions. During forward inference at each layer, the router dynamically selects the top K most relevant experts (Top-K) based on the features of each input token to participate in the computation. This allows the MoE model to linearly scale while only incurring a sublinear increase in computational overhead. The MoE model achieves a certain balance between knowledge volume (parameter scale) and inference efficiency, and therefore has been widely used in recent years. New generation models of large language models such as DeepSeek, Qwen, and LLaMA (a generative text large language model released by Meta) have all adopted the MoE architecture.

[0004] Due to the sparse activation characteristics of MoE models, the total number of parameters has increased rapidly in recent years. For example, the DeepSeek-R1 model (R1 being the version number) has a total of 671 billion parameters. Even when deployed with FP8 (8-bit floating-point) precision, which has a relatively small footprint, it requires 16 NVIDIA H100 GPUs (graphics processing units) across two computing nodes using Expert Parallelism (EP) to support its inference deployment. However, when using Expert Parallelism for MoE model inference, all experts in each layer of the model are equally distributed across all participating GPUs. At this time, the tokens will go to the GPU where the activating expert is located according to the routing mechanism. Therefore, significant all-to-all set communication overhead caused by token swapping is introduced between GPUs, resulting in reduced GPU computing resource utilization during inference. Meanwhile, there is an uneven distribution of the number of terms received by experts across different GPUs. This leads to GPUs with fewer received terms being forced to complete their expert calculations only under the synchronization barrier of all-to-all set communication, further wasting GPU hardware computing resources. On the other hand, while the MoE model is computationally sparse, its model parameters need to be fully loaded into the GPU's memory during inference. This results in small batch sizes for large models, impacting processing efficiency. Experiments show that in parallel expert inference of the DeepSeek-R1 model on two servers equipped with 16 NVIDIA H100 GPUs, the overall utilization of GPU computing resources is less than 30%, which is low.

[0005] Therefore, in order to address these challenges, it is necessary to propose a new parallel inference scheme for hybrid expert models. This scheme aims to efficiently overlap the large communication overhead caused by model computation and expert parallelism in an asynchronous manner, eliminate the waste of computing resources caused by uneven expert participation and synchronous communication, and effectively increase the batch size in the system inference process, thereby improving the overall efficiency of the MoE model inference system.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide an asynchronous parallel inference method, system, device, and medium for hybrid expert models, which can effectively improve the problem of low GPU computing utilization in parallel inference of MoE models, thereby improving the efficiency of the system during inference.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] An asynchronous parallel inference system based on a hybrid expert model includes:

[0010] The asynchronous computation and communication overlay module is used to convert the all-to-all synchronous communication in expert parallelism into point-to-point asynchronous communication, and to combine a multi-threading mechanism to realize the parallel overlay of hybrid expert model computation and word data communication; wherein, the hybrid expert model computation is scheduled and controlled by the scheduling instructions output by the scheduling execution module.

[0011] The data loading module is used in the offline phase to pre-run a portion of the data to be processed, analyze the distribution of hot and cold experts in each layer of the hybrid expert model, and analyze the maximum proportion of cold experts loaded onto CPU memory. In the online phase, for each layer of the hybrid expert model, based on the corresponding distribution of hot and cold experts and the maximum proportion of cold experts loaded, cold experts are loaded onto CPU memory, while the remaining experts are retained in GPU memory. For cold experts loaded onto CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert calculation. Here, GPU is the graphics processing unit, CPU is the central processing unit, and the distribution of hot and cold experts is measured by the probability of experts receiving words. Experts with higher probabilities are more likely to receive words and are called hot experts, while experts with lower probabilities are less likely to receive words and are called cold experts. The expert calculation is part of the hybrid expert model calculation.

[0012] The scheduling execution module is used to record the accumulation of lexical units in different model layers on the GPU and generate scheduling instructions for controlling the computation of hybrid expert models; and to record the loading of cold experts and generate scheduling instructions for controlling the loading of cold experts.

[0013] An asynchronous parallel inference method based on a hybrid expert model, based on the aforementioned system implementation, includes:

[0014] The all-to-all synchronous communication in expert parallelism is converted into point-to-point asynchronous communication, and a multi-threading mechanism is combined to realize the parallel superposition of hybrid expert model computation and word data communication; wherein, the hybrid expert model computation is scheduled and controlled by the scheduling instructions output by the scheduling execution module.

[0015] In the offline phase, a portion of the data to be processed is pre-run to analyze the distribution of hot and cold experts in each layer of the hybrid expert model, as well as the maximum proportion of cold experts loaded onto CPU memory. In the online phase, for each layer of the hybrid expert model, based on the corresponding distribution of hot and cold experts and the maximum proportion of cold experts loaded onto CPU memory, cold experts are loaded onto CPU memory, while the remaining experts are retained in GPU memory. For cold experts loaded onto CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert computation. Here, GPU is the graphics processing unit, and CPU is the central processing unit. The distribution of hot and cold experts is measured by the probability that an expert receives a word. Experts with higher probabilities are more likely to receive words and are called hot experts, while experts with lower probabilities are less likely to receive words and are called cold experts. The expert computation is part of the hybrid expert model computation.

[0016] Record the lexical accumulation in different model layers on the GPU and generate scheduling instructions for controlling the computation of hybrid expert models; and record the loading of cold experts and generate scheduling instructions for controlling the loading of cold experts.

[0017] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0020] As can be seen from the technical solution provided by the present invention, the synchronization of computation and communication between GPUs caused by All-to-all set communication in current expert parallelism is decoupled, allowing asynchronous parallelism between MoE model computation and term data communication on the GPU. This effectively masks the data communication overhead caused by expert parallelism and eliminates synchronization waiting overhead. Based on the common phenomenon of uneven distribution of hot and cold experts within layers in MoE model inference, hot experts with more received terms are preferentially placed in GPU memory, while cold experts with fewer received terms are loaded in CPU memory, thereby freeing up GPU memory space. This allows the MoE model to improve GPU computational efficiency by increasing the batch size during inference. Efficient resource scheduling is achieved by dynamically selecting the most suitable computational unit for execution and the cold experts that need to be loaded. Overall, the solution of the present invention can significantly reduce the communication overhead of the MoE model during expert parallel inference, while improving GPU computational efficiency and optimizing the overall throughput performance during inference. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an asynchronous parallel inference system based on a hybrid expert model, provided as an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the architecture of an asynchronous parallel inference system based on a hybrid expert model, provided as an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the MoE model computation process in an existing parallel inference system provided for an embodiment of the present invention.

[0025] Figure 4 This is a schematic diagram of the model calculation process within the asynchronous computing and communication overlay module provided in an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram illustrating the workflow of the data side-loading module provided in this embodiment of the invention during the online phase of the system.

[0027] Figure 6 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0029] First, the following explanations are provided for the terms that may be used in this article:

[0030] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0031] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0032] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0033] The following provides a detailed description of an asynchronous parallel inference method, system, device, and medium for a hybrid expert model provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of the instruments used in the embodiments of this invention are not specified, they are all conventional products that can be purchased commercially.

[0034] Example 1

[0035] This invention provides an asynchronous parallel inference system based on a hybrid expert model, mainly comprising:

[0036] The asynchronous computation and communication overlay module is used to convert the all-to-all synchronous communication in expert parallelism into point-to-point asynchronous communication, and combine it with a multi-threading mechanism to realize the parallel overlay of hybrid expert model computation and term data communication; wherein, the hybrid expert model computation is scheduled and controlled by the scheduling instructions output by the scheduling execution module.

[0037] The data loading module is used to pre-run a portion of the data to be processed during the offline phase, analyze the distribution of hot and cold experts in each layer of the hybrid expert model, and analyze the maximum cold expert loading ratio loaded onto CPU memory. During the online phase, in each layer of the hybrid expert model, based on the corresponding hot and cold expert distribution and the maximum cold expert loading ratio, cold experts are loaded onto CPU memory, while the remaining experts are retained in GPU memory. For cold experts loaded onto CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert calculation. Here, GPU is the graphics processing unit, and CPU is the central processing unit. The distribution of hot and cold experts is measured by the probability that an expert receives a word. Experts with higher probabilities are more likely to receive words and are called hot experts, while experts with lower probabilities are less likely to receive words and are called cold experts. The expert calculation is part of the hybrid expert model calculation.

[0038] The scheduling execution module is used to record the accumulation of lexical units in different model layers on the GPU and generate scheduling instructions (computation scheduling) for controlling the computation of hybrid expert models; and to record the side loading of cold experts and generate scheduling instructions (side loading scheduling) for controlling the loading of cold experts.

[0039] The term accumulation mentioned here refers to the cumulative number of terms dynamically gathered on the corresponding computing units during the reasoning process, which can be obtained through conventional techniques and will not be elaborated upon in this invention.

[0040] The method provided in the embodiments of the present invention has the following main advantages:

[0041] (1) Decouple the synchronization of computation and communication between GPUs caused by All-to-all set communication in the current expert parallelism, and allow the asynchronous parallelism between MoE model computation and word data communication on the GPU, thereby fully masking the data communication overhead caused by expert parallelism and eliminating the synchronization waiting overhead.

[0042] (2) Based on the phenomenon of uneven distribution of hot and cold experts within the layer in the MoE model inference, hot experts with more received words are placed in the GPU memory first, while cold experts with fewer received words are placed in the CPU memory, thereby freeing up GPU memory space. This allows the MoE model to improve GPU computational efficiency by increasing the batch size during inference.

[0043] (3) By dynamically selecting the most suitable computing unit (including attention unit and hybrid expert unit, which will be explained in detail later) and the cold experts that need to be loaded, efficient resource scheduling is achieved.

[0044] Overall, the solution of this invention can significantly reduce the communication overhead of the MoE model during expert parallel inference, while improving the computational efficiency of the GPU and optimizing the overall throughput performance during the inference process.

[0045] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0046] I. Overall Introduction to the Plan.

[0047] This invention provides an asynchronous parallel inference method for hybrid expert models, aiming to improve the overall inference efficiency when using expert parallelism for inference. To achieve this goal, the method provided by this invention mainly includes three modules: an asynchronous computation and communication overlay module, a data loading module, and a scheduling and execution module, such as... Figure 1 As shown.

[0048] In the asynchronous computation and communication overlay module, the synchronization of computation and communication between GPUs caused by All-to-all set communication in the current expert parallelism is decoupled, enabling asynchronous model computation and word data communication on the GPU, thereby reducing GPU idle time, masking the data communication overhead caused by parallelism, and eliminating synchronization waiting overhead. Specifically, the All-to-all set communication of word exchange is decoupled into asynchronous point-to-point Send-Recv communication between each GPU, and a multi-threaded mechanism is used to split the implementation of model computation and data communication into multiple threads for parallel execution, enabling full overlay of computation and data communication.

[0049] In the data sideloading module, addressing the common issue of uneven distribution of hot and cold experts within layers during MoE model inference, hot experts with a high number of received terms are prioritized for placement in GPU memory, while cold experts with fewer received terms are sideloaded in CPU memory. This frees up GPU memory, allowing the model to improve computational efficiency during inference by increasing the batch size. Specifically, in the offline phase before MoE model inference, the distribution of hot and cold experts at each layer is obtained through pre-running on a dataset of a certain size. Subsequently, based on this distribution, a portion of cold experts are sideloaded to CPU memory, freeing up GPU memory to increase the maximum batch size that the system can accommodate during inference. For the sideloaded cold experts, they await scheduling back to the GPU for execution in the online phase, based on the accumulation of their received terms.

[0050] In the scheduling and execution module, scheduling information such as the lexical accumulation and cold expert side-loading status of each computing unit in different model layers on the GPU is recorded. Based on the scheduling information, the most suitable computing unit for execution and the reloaded cold expert are selected under the guidance of the scheduling algorithm. Then, the relevant processes in the first two modules are controlled through scheduling instructions. Specifically, considering the computing resources, storage resources, and communication resources in the GPU, scheduling instructions for controlling the computation of the hybrid expert model are generated in the direction of improving GPU utilization, based on the lexical accumulation status of each computing unit, and input into the asynchronous computing and communication overlay module. At the same time, the lexical accumulation progress of the cold experts is monitored. Based on masking the side-loading overhead (i.e., the time for the cold expert to be loaded back to the GPU is less than the current computation time of the hybrid expert model, and the execution time will not be extended due to waiting for the cold expert to be loaded), the cold expert with the highest accumulation level is selected. This generates scheduling instructions for controlling the loading of the corresponding cold expert in the CPU back to the GPU, and is input into the data side-loading module.

[0051] Based on the above three modules, not only does asynchronous expert parallelism effectively mask the communication overhead and waiting overhead caused by a large amount of word synchronous data communication in the MoE model, but it also effectively improves the computing efficiency of the GPU by increasing the batch size through cold expert data side loading, providing a more efficient and economical solution for the parallel inference system of the MoE model.

[0052] II. Detailed introduction of the plan.

[0053] Figure 2 The detailed architecture of the asynchronous parallel inference method for hybrid expert models provided in this embodiment of the invention is illustrated, and each part is described in detail below.

[0054] 1. Asynchronous computing and communication overlay module.

[0055] In this invention, the asynchronous computation and communication overlay module aims to achieve asynchronous expert parallel inference. It changes the execution mode of MoE model inference from inference layer by layer at the model layer granularity to execution at the unit granularity within the model layer according to the scheduling order. This allows requests within an inference batch to have different execution progress and eliminates the synchronous dependency between requests for computation and requests for lexical communication. Subsequently, through a multi-threaded mechanism, the main thread is responsible for the computation and execution of the model, while the communication thread is responsible for the communication of lexical data, enabling efficient overlay of model computation and data communication during the inference process.

[0056] To intuitively understand the innovation of this invention, the following section will introduce the parallel techniques used in parallel inference of the MoE model.

[0057] Figure 3This demonstrates the computational flow of the existing MoE model during parallel inference. The MoE model also features the repetitive layer structure characteristic of Transformer-like models. Each layer primarily consists of an attention layer (Attention layer) for attention computation and a hybrid expert layer (MoE layer) containing multiple experts and routes. Here, the attention layer and the routes in the hybrid expert layer are divided into attention units (Attention units), responsible for attention computation and routing; the part containing only expert computation in the hybrid expert layer is divided into hybrid expert units (MoE units), responsible for expert computation. When deploying the MoE model for parallel inference, current popular solutions (such as the open-source inference framework SGLang) often employ different parallel strategies for different units in the model: data parallelism (DP) is used for attention units with a small parameter percentage, copying the same parameters to the GPU memory involved in the computation, allowing each GPU to independently accept input requests; expert parallelism is used for hybrid expert units with a large parameter percentage, evenly distributing the experts in each hybrid expert unit across the GPUs involved in the computation, reducing memory pressure.

[0058] When using the parallel scheme described above for inference of the MoE model, each GPU has a corresponding execution process on the CPU. The main thread within each execution process receives input requests and performs independent computation in the attention units of each layer until the routing results are obtained to determine the Top-K experts for each term in the request that will be processed by the hybrid expert unit. Subsequently, each GPU initiates all-to-all synchronous communication, sending the term data located in the memory of other GPUs to the corresponding GPU, while simultaneously receiving the term data sent to itself. After all GPUs have completed communication, the GPU uses its own experts to perform expert computation on the received terms. After the expert computation is completed, the GPU initiates all-to-all communication again to restore the terms contained in the request before entering the hybrid expert unit computation, aggregating the term results computed on the experts of different GPUs, and then processing them layer by layer in the subsequent model until the model is executed to the end and output is generated. However, all-to-all communication of word data between GPUs causes a lot of communication overhead. At the same time, due to the different hot and cold nature of experts on different GPUs, there is a phenomenon that GPUs with fewer received words and shorter expert calculation times are idle and waiting for GPUs with more received words and longer expert calculation times to complete the all-to-all word reconstruction communication before proceeding with subsequent calculations, which further reduces the utilization of GPUs.

[0059] To address the low GPU utilization issue in existing expert parallelism based on all-to-all synchronous communication, this invention proposes an asynchronous computation and communication overlay module in the expert parallel inference method, such as... Figure 4 As shown. The asynchronous computation and communication overlay module changes the execution method during model inference from model-layer granularity, layer-by-layer inference to unit-level granularity within the model layer, based on the scheduling order. This allows requests within a batch to have different execution schedules, creating opportunities for the overlay of model computation and communication overhead. For example, after executing the first... After the attention unit of a layer is computed, it may execute hybrid expert units of other model layers according to scheduling instructions. During this process, the first layer is completed. The terminology data communication of the layered hybrid expert unit is then performed, followed by the execution of the next step. The computation of the hybrid expert unit at the layer enables the system to perform the computation of the first layer. When requesting tokens for communication within a layer, no synchronous waiting is required, and model computations in other layers can be invoked to mask the communication overhead. During data communication, the all-to-all synchronous communication for token exchange is completely transformed into point-to-point Send-Recv communication between each GPU, further eliminating synchronization and increasing the asynchronous space.

[0060] Specifically, a multi-threaded mechanism is used to achieve asynchronous superposition of model computation and term data communication in expert parallelism. Each GPU has a corresponding execution process on the CPU, which includes a main thread and a communication thread. The main thread is the computation thread, which continuously launches CUDA (Unified Computing Device Architecture) kernel functions on the GPU to perform computations on computational units within different layers of the hybrid expert model. These computational units include attention units and hybrid expert units. Simultaneously, the communication thread on the execution process exchanges term data with the GPU containing the object experts in the routing results, based on the routing results calculated by different attention units, using point-to-point asynchronous communication.

[0061] In the embodiments of this invention, all GPUs involved are GPUs participating in expert parallelism, which will not be elaborated upon in the text.

[0062] In this embodiment of the invention, point-to-point asynchronous communication can be a two-stage point-to-point notification-transmission communication, with the exchange of metadata achieved through the NCCL Send-Recv (NVIDIA Collective Communication Library point-to-point send-receive) communication primitive. In this way, the main thread responsible for computation and the sub-thread responsible for communication can run simultaneously, fully realizing the superposition of computational and communication overhead.

[0063] In this embodiment of the invention, the point-to-point notification-transmission two-stage communication is implemented as follows: Each GPU maintains a fixed message queue in the communication thread within the corresponding execution process on the CPU. The communication thread within the execution process of each GPU acts as either a data receiver or a data sender during the communication process. When the data sender obtains the lexical routing information, it notifies the data receiver's message queue via the CPU to send the metadata of the data to be transmitted, waiting for the data receiver to retrieve the metadata from the message queue, including information such as the data sender's information and the amount of data. After the data sender and data receiver complete the exchange of metadata information, they respectively start non-blocking NCCL Send and NCCL Recv communication primitives on their respective GPUs to transmit lexical metadata in the background. Simultaneously, the data receiver and data sender can also send or receive other lexical communication tasks without waiting for the current lexical communication task to finish. Here, NCCL Send is a point-to-point send function of the NVIDIA Collective Communication Library, and NCCL Recv is a point-to-point receiver function of the NVIDIA Collective Communication Library.

[0064] 2. Data loading module.

[0065] In this invention embodiment, the data sideloading module includes: In the offline phase before formal inference, it pre-runs to obtain the layer-by-layer distribution of hot and cold experts in the MoE model for the current task, as well as the maximum cold expert sideloading ratio whose sideloading overhead can be masked. Upon entering the online phase, during the model weight loading phase, according to the hot and cold expert distribution and the maximum cold expert sideloading ratio obtained in the offline phase, the coldest experts are sideloaded to the CPU layer by layer until the maximum cold expert sideloading ratio is achieved, thereby freeing up more GPU memory space to increase the batch size during inference and improve GPU computational utilization. During the inference phase, the system executes computational units in different layers in the asynchronous computation and communication overlay module according to the schedule. For the cold experts sideloaded in each layer of mixed expert units, the data sideloading module will, during the inference process, load them back to the GPU for computation as needed, based on their lexical accumulation level, while masking the loading overhead.

[0066] In this embodiment of the invention, during the offline stage before formal inference, the MoE model is pre-run on a small dataset. The distribution of cold and hot experts in the model layer by layer and the maximum cold expert sideload ratio loaded onto the CPU memory are obtained through the cold and hot analyzer and sideload solver in the data sideloading module.

[0067] (1) Cold and hot analyzer.

[0068] The hot and cold expert analyzer inserts a dictionary-like data structure after the routing mechanism of each layer of the hybrid expert model. While pre-running a portion of the data to be processed, it captures the Top-K experts to which each processing request's tokens are routed, and counts these experts in the corresponding records within the dictionary-like data structure. The Top-K experts are the K experts with the highest relevance to the token features. Then, the count results recorded in the dictionary-like data structure of each layer of the hybrid expert model are processed using a normalized exponential (Softmax) function to convert the absolute number of tokens received by each expert into the probability of each token being received, obtaining the probability distribution of expert token reception. This probability distribution is used as the hot and cold expert distribution. This probability distribution, reflecting the hot and cold nature of experts, serves as a reference for selecting side-loaded experts during the model weight loading process in the online phase.

[0069] (2) Side load solver.

[0070] The sideload solver obtains information on how available computing resources and communication performance vary with the amount of data being processed by running a series of benchmark tests.

[0071] For data communication between the GPU and CPU (i.e., loading cold expert data), multiple data sizes are set for transmission, and after sampling the communication delay, the following is used: The model is used to fit the communication latency performance under different data volumes. The model is represented as:

[0072] ;

[0073] in, The time required to transmit a message of size m bytes. A fixed delay is set for initiating communication. It represents the data transmission time per byte, which is equivalent to the reciprocal of the communication bandwidth.

[0074] For matrix multiplication, attention computation, and expert computation on GPUs, after sampling the computation latency using multiple batch sizes of input, linear fitting was used between each pair of adjacent batch size sampling points to simulate the computation latency of different types of computations under different batch sizes, serving as performance analysis data. Subsequently, based on the performance analysis data obtained from the tests, the relationship between the side-load ratio of each layer of experts in the hybrid expert model and the maximum batch size was modeled, and the maximum proportion of loading overhead that could be masked by the single-layer computation execution time of the hybrid expert model was selected as the maximum cold expert side-load ratio.

[0075] Specifically: Based on the communication latency variation between the GPU and CPU as the amount of data transmitted changes, a loading overhead is constructed for reloading a layer of cold experts back to the GPU when a certain proportion of experts are loaded. Simultaneously, considering the variation of computational latency for matrix multiplication, attention calculation, and expert calculation at each layer of the hybrid expert model on the GPU with batch size, the execution time of each layer of the model is constructed based on the maximum batch size supported by the GPU memory when the sideload is set to a certain proportion of cold experts. This ensures that the following formula always holds true:

[0076] ;

[0077] And the maximum cold expert side load ratio is solved under the condition that the formula holds.

[0078] Based on the output of the thermal analyzer and the side load solver, the working mode during the online phase is as follows: Figure 5 As shown. The online phase includes a weight loading phase and an inference phase. In the weight loading phase, for each layer of the hybrid expert model, the parameters in the attention unit are fully loaded onto all GPU memory involved in inference. The parameters in the hybrid expert unit (i.e., all experts included in the hybrid expert unit) are evenly distributed across each GPU memory involved in inference based on the number of GPUs. On each GPU, based on the distribution of hot and cold experts, the coldest experts are sideloaded onto CPU memory starting with the coldest experts until the maximum sideloading ratio of cold experts is reached. The remaining experts are retained in GPU memory. Afterwards, a data buffer the size of the sideloaded experts within a layer is allocated on the GPU to store the cold experts loaded back onto GPU memory for computation. In the inference phase, the terms routed to cold experts during inference cannot be immediately computed and are temporarily stored in the GPU for term count accumulation. The data loading module loads cold experts back to the GPU memory for expert calculation at the layer level, based on the degree of lexical accumulation on the cold experts being loaded, under the control of scheduling instructions. The entire loading process uses non-blocking PCIe (high-speed serial computer expansion bus standard) data transmission to ensure that the loading overhead is masked by the calculation.

[0079] 3. Scheduling and execution module.

[0080] In this invention, the scheduling execution module records the lexical accumulation of each computing unit in different model layers on the GPU and the scheduling information of the cold expert side loading. Based on the scheduling information, the most suitable computing unit for execution and the reloaded cold expert are selected under the guidance of the scheduling algorithm. Then, scheduling instructions are generated and applied to the two schemes mentioned above.

[0081] Specifically, each GPU has a corresponding unit scheduler located in the main thread of the execution process on the CPU corresponding to the GPU. This scheduler uses a dictionary-like data structure to track the location of each request in the batch of inference requests, such as the accumulated cache of attention units or hybrid expert units in a specific layer. It also records the numbers of cold experts sideloaded to the CPU in each layer of the model, forming the scheduling information referenced during scheduling. When computation on the GPU is complete, the scheduling algorithm within the unit scheduler refers to this information to select the appropriate computational unit for the GPU to execute, preventing the GPU from becoming idle and maintaining its high-efficiency computation. Simultaneously, the scheduling algorithm considers the impact of the tokens accumulated on the sideloaded cold experts on the available GPU memory, loading them back to the GPU for computation while masking the overhead of loading cold experts.

[0082] The scheduling algorithm prioritizes executing hybrid expert units with a large number of accumulated terms in the cache, followed by attention units with a large number of terms. This ensures that units with large batches of terms are processed first, maximizing GPU utilization. If too many terms are blocked by cold experts, causing requests for those terms to occupy too much KV (key-value data) cache and exceeding the threshold that would affect normal system operation, the cold experts in the model layer with the largest number of accumulated terms are loaded back while the GPU is performing computation. They are then scheduled to perform computation, releasing the blocked terms to subsequent units and promoting the early release of the KV cache.

[0083] In the implementation of the above-mentioned solution provided by the embodiments of the present invention, each module can be deployed on the CPU. During the inference process, the CPU controls each module to submit computation, communication or cold expert loading tasks to the GPU.

[0084] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0085] Example 2

[0086] This invention also provides an asynchronous parallel inference method based on a hybrid expert model, which is mainly based on the system implementation provided in the foregoing embodiments. The method mainly includes:

[0087] The all-to-all synchronous communication in expert parallelism is converted into point-to-point asynchronous communication, and a multi-threading mechanism is combined to realize the parallel superposition of hybrid expert model computation and word data communication; wherein, the hybrid expert model computation is scheduled and controlled by the scheduling instructions output by the scheduling execution module.

[0088] In the offline phase, a portion of the data to be processed is pre-run to analyze the distribution of hot and cold experts in each layer of the hybrid expert model, as well as the maximum proportion of cold experts loaded onto CPU memory. In the online phase, for each layer of the hybrid expert model, based on the corresponding distribution of hot and cold experts and the maximum proportion of cold experts loaded onto CPU memory, cold experts are loaded onto CPU memory, while the remaining experts are retained in GPU memory. For cold experts loaded onto CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert computation. Here, GPU is the graphics processing unit, and CPU is the central processing unit. The distribution of hot and cold experts is measured by the probability that an expert receives a word. Experts with higher probabilities are more likely to receive words and are called hot experts, while experts with lower probabilities are less likely to receive words and are called cold experts. The expert computation is part of the hybrid expert model computation.

[0089] Record the lexical accumulation in different model layers on the GPU and generate scheduling instructions for controlling the computation of hybrid expert models; and record the loading of cold experts and generate scheduling instructions for controlling the loading of cold experts.

[0090] Since the main technical details involved in this method have been described in detail in the previous embodiments, they will not be repeated here.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0092] Example 3

[0093] The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0094] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0095] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0096] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0097] The output device can be a display terminal;

[0098] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0099] Example 4

[0100] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0101] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0102] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. An asynchronous parallel inference system using a hybrid expert model, characterized in that, include: The asynchronous computation and communication overlay module is used to convert the all-to-all synchronous communication in expert parallelism into point-to-point asynchronous communication, and to combine a multi-threading mechanism to realize the parallel overlay of hybrid expert model computation and word data communication; wherein, the hybrid expert model computation is scheduled and controlled by the scheduling instructions output by the scheduling execution module. The data loading module is used in the offline phase to pre-run a portion of the data to be processed, analyze the distribution of hot and cold experts in each layer of the hybrid expert model, and analyze the maximum proportion of cold experts loaded onto CPU memory. In the online phase, for each layer of the hybrid expert model, based on the corresponding distribution of hot and cold experts and the maximum proportion of cold experts loaded, cold experts are loaded onto CPU memory, while the remaining experts are retained in GPU memory. For cold experts loaded onto CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert calculation. Here, GPU is the graphics processing unit, CPU is the central processing unit, and the distribution of hot and cold experts is measured by the probability of experts receiving words. Experts with higher probabilities are more likely to receive words and are called hot experts, while experts with lower probabilities are less likely to receive words and are called cold experts. The expert calculation is part of the hybrid expert model calculation. The scheduling execution module is used to record the accumulation of lexical units in different model layers on the GPU and generate scheduling instructions for controlling the computation of hybrid expert models; and to record the loading of cold experts and generate scheduling instructions for controlling the loading of cold experts.

2. The asynchronous parallel inference system based on a hybrid expert model according to claim 1, characterized in that, The parallel superposition of hybrid expert model computation and term data communication using a multi-threaded mechanism includes: Each layer in the hybrid expert model contains an attention unit and a hybrid expert unit. The attention unit is responsible for attention calculation and routing, while the hybrid expert unit is responsible for expert calculation. Each GPU has a corresponding execution process on the CPU, which includes a main thread and a communication thread. The main thread is a computation thread that continuously launches CUDA kernel functions on the GPU to perform computations on computational units within different layers of the hybrid expert model. CUDA is a unified computing device architecture, and the computational units include attention units and hybrid expert units. Meanwhile, the communication thread on the execution process exchanges term data with the GPU containing the object experts in the routing results based on the routing results calculated by different attention units, using point-to-point asynchronous communication.

3. The asynchronous parallel inference system based on a hybrid expert model according to claim 2, characterized in that, The point-to-point asynchronous communication is a two-phase communication of point-to-point notification and transmission, and is implemented as follows: Each GPU maintains a fixed message queue in the communication thread within the execution process corresponding to the CPU. The communication thread within the execution process corresponding to each GPU acts as either a data receiver or a data sender during the communication process. When the data sender obtains the word routing information, it notifies the data receiver's message queue of the metadata of the data to be transmitted through the CPU. The data receiver waits to retrieve the metadata from the message queue, which includes information such as the data sender and the amount of data. After the data sender and receiver complete the exchange of metadata information, they will each start non-blocking NCCL Send and NCCL Recv communication primitives on their respective GPUs to transmit lexical metadata. At the same time, the data receiver and the data sender can also send or receive other lexical communication tasks. Among them, NCCL Send is a point-to-point send function of the NVIDIA Collective Communication Library, and NCCL Recv is a point-to-point receiver function of the NVIDIA Collective Communication Library.

4. The asynchronous parallel inference system based on a hybrid expert model according to claim 1, characterized in that, The distribution of hot and cold experts in each layer of the analytical hybrid expert model includes: A dictionary-like data structure is inserted after the routing mechanism of each layer of the hybrid expert model. When a portion of the data to be processed is pre-run, the top-K experts to which the lexical tokens are routed in each processing request are captured, and the count is performed at the record corresponding to the expert in the dictionary-like data structure. The top-K experts refer to the top K experts with the highest relevance to the lexical token features. The counting results recorded in each layer of the dictionary-like data structure of the hybrid expert model are processed using a normalized exponential function to convert the absolute number of expert-received words into the probability of expert-received words, thereby obtaining the probability distribution of expert-received words. This probability distribution is then used as the distribution of hot and cold experts.

5. The asynchronous parallel inference system based on a hybrid expert model according to claim 1, characterized in that, The maximum cold expert load ratio that the analysis side loads onto CPU memory includes: A series of benchmark tests were run to obtain the changes in available computing resources and communication performance with the amount of data processed; for data communication between the GPU and CPU, communication latency was sampled after setting multiple data sizes, and then... The model is used to fit the communication latency performance under different data volumes. The model is represented as: ； in, The time required to transmit a message of size m bytes. A fixed delay is set for initiating communication. The data transmission time per unit byte is equivalent to the reciprocal of the communication bandwidth. For matrix multiplication, attention computation, and expert computation on GPUs, after sampling the computation latency for inputs with multiple batch sizes, linear fitting is used between each pair of adjacent batch size sampling points to simulate the computation latency of different types of computations under different batch sizes, which serves as performance analysis data. Subsequently, based on the performance analysis data obtained from the test, the relationship between the sideload ratio of each layer of experts in the hybrid expert model and the maximum batch size was modeled. The maximum ratio of loading overhead that could be masked by the single-layer computation execution time of the hybrid expert model was selected as the maximum cold expert sideload ratio. This included: based on the change in communication latency between the GPU and CPU as the amount of data transmitted, constructing the loading overhead of reloading a layer of cold experts back to the GPU memory when the sideload ratio of experts is set. Simultaneously, considering the variation of computational latency for matrix multiplication, attention calculation, and expert calculation at each layer of the hybrid expert model on the GPU with batch size, the execution time of each layer of the model is constructed based on the maximum batch size supported by the GPU memory when the sideload is set to a certain proportion of cold experts. This ensures that the following formula always holds true: ； And the maximum cold expert side load ratio is solved under the condition that the formula holds.

6. The asynchronous parallel inference system based on a hybrid expert model according to claim 1, characterized in that, In the online phase, each layer of the hybrid expert model combines the corresponding distribution of hot and cold experts with the maximum cold expert sideload ratio, sideloads the cold experts to the CPU memory, and retains the remaining experts in the GPU memory. For cold experts that are loaded into CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert calculations, including: The online phase includes: the weight loading phase and the inference phase; During the weight loading phase, each layer of the hybrid expert model contains an attention unit and a hybrid expert unit. The attention unit is responsible for attention calculation and routing, while the hybrid expert unit is responsible for expert calculation. The parameters in the attention unit are fully loaded onto the GPU memory of all participating in inference. The parameters in the hybrid expert unit are all the experts contained in the hybrid expert unit. Based on the number of GPUs, the experts are evenly distributed across the GPU memory of each participating in inference. On each GPU, based on the distribution of hot and cold experts, the coldest experts are sideloaded onto the CPU memory starting from the coldest experts until the maximum sideloading ratio of cold experts is reached. The remaining experts are kept in the GPU memory. Afterward, a data cache the size of the sideloaded experts within the layer is allocated on the GPU to store the cold experts loaded back onto the GPU for calculation later. During the inference phase, the terms routed to the cold experts during inference will be temporarily stored in the GPU to accumulate the term count. The data loading module will load the cold experts back to the GPU memory at the layer level according to the degree of term accumulation on the loaded cold experts and under the control of the scheduling instructions, for expert calculation.

7. The asynchronous parallel inference system based on a hybrid expert model according to claim 1, characterized in that, The system records the accumulation of lexical units in different model layers on the GPU and generates scheduling instructions for controlling the computation of hybrid expert models. And it records the load status of the cooling expert and generates scheduling instructions for controlling the loading of the cooling expert, including: By integrating the computing, storage, and communication resources of the GPU, and based on the word accumulation of the computing units in each layer, scheduling instructions for controlling the computation of the hybrid expert model are generated in the direction of improving GPU utilization, and input into the asynchronous computing and communication overlay module. Each layer in the hybrid expert model contains two types of computing units: attention units and hybrid expert units. Attention units are responsible for attention calculation and routing, while hybrid expert units are responsible for expert calculation. The algorithm monitors and records the lexical accumulation progress of cold experts, selects the cold expert with the highest accumulation level based on the masking side loading overhead, and generates a scheduling instruction to control the loading of the corresponding cold expert from the CPU back to the GPU memory, which is then input to the data side loading module. Here, the masking side loading overhead refers to the time it takes for the cold expert to be loaded back to the GPU being less than the time it takes for the current hybrid expert model to be computed.

8. An asynchronous parallel reasoning method using a hybrid expert model, characterized in that, The system implementation based on any one of claims 1 to 7 includes: The all-to-all synchronous communication in expert parallelism is converted into point-to-point asynchronous communication, and a multi-threading mechanism is combined to realize the parallel superposition of hybrid expert model computation and word data communication; wherein, the hybrid expert model computation is scheduled and controlled by the scheduling instructions output by the scheduling execution module. In the offline phase, a portion of the data to be processed is pre-run to analyze the distribution of hot and cold experts in each layer of the hybrid expert model, as well as the maximum proportion of cold experts loaded onto CPU memory. In the online phase, for each layer of the hybrid expert model, based on the corresponding distribution of hot and cold experts and the maximum proportion of cold experts loaded onto CPU memory, cold experts are loaded onto CPU memory, while the remaining experts are retained in GPU memory. For cold experts loaded onto CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert computation. Here, GPU is the graphics processing unit, and CPU is the central processing unit. The distribution of hot and cold experts is measured by the probability that an expert receives a word. Experts with higher probabilities are more likely to receive words and are called hot experts, while experts with lower probabilities are less likely to receive words and are called cold experts. The expert computation is part of the hybrid expert model computation. Record the lexical accumulation in different model layers on the GPU and generate scheduling instructions for controlling the computation of hybrid expert models; and record the loading of cold experts and generate scheduling instructions for controlling the loading of cold experts.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in claim 8.

10. A readable storage medium storing a computer program, characterized in that, The method described in claim 8 is implemented when the computer program is executed by the processor.

Images