Asynchronous parallel reasoning method, system and equipment for hybrid expert model and medium

Abstract

The invention discloses an asynchronous parallel reasoning method, system and equipment for a hybrid expert model and a medium, which are corresponding schemes: decoupling synchronization of calculation and communication between GPUs (Graphics Processing Unit) caused by all-to-all set communication in expert parallelism, allowing asynchronous parallelism of model calculation and lexical metadata communication, and solving the problem of asynchronous parallelism of the model calculation and lexical metadata communication. Data communication overhead caused by expert parallelization is fully masked, and synchronization waiting overhead is eliminated; aiming at the phenomenon of uneven cold and heat of experts in reasoning, the hot experts are preferentially placed in the GPU, the cold experts are laterally loaded in the CPU so as to release the video memory space of the GPU, and the calculation efficiency of the GPU can be improved by increasing the batch size during reasoning; efficient resource scheduling is realized by dynamically selecting a computing unit which is most suitable for execution and a cold expert which needs to be loaded; generally speaking, the communication overhead and the waiting overhead during parallel reasoning of experts can be remarkably reduced, meanwhile, the calculation efficiency of the GPU is improved, and the overall throughput performance in the reasoning process is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mixed expert model reasoning, and particularly relates to a mixed expert model asynchronous parallel reasoning method, system, device and medium. BACKGROUND

[0002] In recent years, the deep neural network structure represented by the Transformer model has triggered a profound revolution in the field of natural language processing. The core innovation of the Transformer is its attention mechanism, which models the dependency between positions in the input sequence through linear modeling, enabling efficient parallel matrix computation and handling long-distance dependencies. This architecture has greatly promoted the development of large-scale pre-training models and has become the basis for many natural language generation and understanding models.

[0003] As the size of large language models continues to expand, the traditional dense architecture (Dense Model) has significantly improved the parameter quantity and expression ability, but also brought huge computational overhead, because it needs to process all parameters in the linear expansion feed-forward network (FFN) layer of each layer of the model for each token in the input. To alleviate the demand for computing resources in the training and inference process of large language models, the academic and industrial communities have begun to actively explore sparse structures in recent years, and the Mixture of Experts (MoE) architecture is a typical representative. In the MoE architecture, the FFN layer is replaced by the MoE layer, which is composed of multiple small linear matrices called experts and an activation function. During the forward inference of each layer, the router mechanism selects the top-K experts with the highest relevance according to the characteristics of each input token to participate in the calculation, which allows the MoE model to linearly expand the model size while only bringing sub-linear growth in computational overhead. The MoE model achieves a certain balance between knowledge quantity (parameter size) and inference efficiency, so it has been widely used in recent years, such as the new generation of large language models such as DeepSeek, Qwen, and LLaMA (Meta's generative text large language model), which all adopt the MoE architecture.

[0004] The MoE model has a rapidly increasing total number of parameters due to its sparse parameter activation characteristics in recent years. For example, the DeepSeek-R1 model (R1 is the version number) has a total of 671B (6710 billion) parameters. Even if it is deployed at a small footprint FP8 (8-bit floating point) precision, it needs to be divided into 16 NVIDIA H100 GPUs (a graphics processor developed by NVIDIA) in 2 computing nodes to support its inference deployment using expert parallelism (EP). However, when using expert parallelism for MoE model inference, all experts in each layer of the model are equally divided among all GPUs participating in expert parallelism. At this time, the word will go to the GPU where the activated expert is located according to the results of the routing mechanism, thus introducing significant all-to-all set communication overhead due to the exchange of words between GPUs, resulting in reduced utilization of GPU computing resources during inference. At the same time, there are also differences in the number of words received by experts divided on different GPUs, which will cause the GPU with fewer total word reception to be forced to wait for the GPU with more word reception to complete the calculation of the expert before proceeding to the subsequent calculation, further wasting GPU hardware computing resources. On the other hand, the MoE model is sparse in computation, but its model parameters need to be completely loaded into the GPU's video memory during inference, which will cause the inference system to have a small batch size when processing requests when the model size is large, affecting processing efficiency. Experiments show that when testing the expert parallel inference of the DeepSeek-R1 model on 2 servers equipped with 16 NVIDIA H100 GPUs, the overall utilization of GPU computing resources is less than 30%, which is at a low level.

[0005] Therefore, in order to address these challenges, it is necessary to propose a new parallel inference scheme for hybrid expert models, which aims to efficiently overlap model computation and large communication overhead caused by expert parallelism in an asynchronous manner, eliminate the waste of computing resources caused by uneven distribution of experts and synchronous communication, and effectively improve the batch size during system inference, thereby improving the efficiency of the MoE model inference system as a whole.

[0006] In view of this, the present application is proposed. SUMMARY

[0007] The purpose of the present application is to provide an asynchronous parallel inference method, system, device and medium for hybrid expert models, which can effectively improve the low GPU computing utilization rate of MoE models during parallel inference, thereby improving the efficiency of the system during inference.

[0008] The object of the application is achieved by the following technical solutions: An asynchronous parallel inference system of a hybrid expert model comprises: An asynchronous computing communication superposition module is configured to convert full-to-full synchronous communication in expert parallelism into point-to-point asynchronous communication, and realize parallel superposition of hybrid expert model computing and word data communication in combination with a multi-thread mechanism; wherein the hybrid expert model computing is scheduled and controlled by scheduling instructions output by a scheduling execution module; A data side loading module is configured to, in an offline stage, pre-run a part of data to be processed, analyze cold and hot expert distribution in each layer of the hybrid expert model, and analyze the maximum cold expert side loading ratio side loaded to CPU memory; in an online stage, in each layer of the hybrid expert model, the cold experts are side loaded to the CPU memory in combination with the corresponding cold and hot expert distribution and the maximum cold expert side loading ratio, and the remaining experts are retained in the GPU memory; for the cold experts side loaded to the CPU memory, they are loaded back to the GPU memory for expert computing in a layer granularity under the control of the scheduling instructions; wherein the GPU is a graphics processing unit, the CPU is a central processing unit, the cold and hot expert distribution is measured by the probability of expert receiving words, the higher the probability, the higher the possibility of the expert receiving words, which is called a hot expert, and the lower the probability, the lower the possibility of the expert receiving words, which is called a cold expert; the expert computing belongs to a part of the hybrid expert model computing; A scheduling execution module is configured to record word accumulation in different model layers on the GPU, and generate scheduling instructions for controlling hybrid expert model computing; and record cold expert side loading, and generate scheduling instructions for controlling cold expert loading.

[0009] An asynchronous parallel inference method of a hybrid expert model, based on the foregoing system, comprises: Converting full-to-full synchronous communication in expert parallelism into point-to-point asynchronous communication, and realizing parallel superposition of hybrid expert model computing and word data communication in combination with a multi-thread mechanism; wherein the hybrid expert model computing is scheduled and controlled by scheduling instructions output by a scheduling execution module; In the offline stage, a part of the data to be processed is run in advance, the hot and cold expert distribution in each layer of the mixed expert model is analyzed, and the maximum cold expert side loading ratio side loaded on the CPU memory is analyzed; in the online stage, in each layer of the mixed expert model, the corresponding hot and cold expert distribution and the maximum cold expert side loading ratio are combined to side load the cold experts to the CPU memory, and the remaining experts are retained in the GPU memory; for the cold experts side loaded to the CPU memory, they are loaded back to the GPU memory for expert calculation under the control of the scheduling instruction; wherein the GPU is a graphics processing unit, the CPU is a central processing unit, the hot and cold expert distribution is measured by the probability of expert receiving tokens, the higher the probability, the higher the possibility of receiving tokens, which is called hot expert, and the lower the probability, the lower the possibility of receiving tokens, which is called cold expert; the expert calculation belongs to a part of the mixed expert model calculation; The token accumulation in different model layers on the GPU is recorded, and scheduling instructions for controlling the mixed expert model calculation are generated; and the cold expert side loading is recorded, and scheduling instructions for controlling the cold expert loading are generated.

[0010] A processing device, comprising: one or more processors; a memory for storing one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.

[0011] A readable storage medium storing a computer program, which implements the foregoing method when executed by a processor.

[0012] As can be seen from the technical solutions provided by the above-mentioned application, the synchronization of GPU inter-computation and communication caused by All-to-all set communication in current expert parallelism is decoupled, allowing MoE model computation and token data communication on the GPU to be performed asynchronously and in parallel, thereby fully masking the data communication overhead caused by expert parallelism, while eliminating the synchronization waiting overhead; according to the phenomenon of uneven distribution of hot and cold experts within layers commonly existing in MoE model inference, hot experts receiving more tokens are placed in the GPU memory, and cold experts receiving fewer tokens are side loaded in the CPU memory, thereby releasing GPU memory space, allowing the MoE model to increase the processing batch size during inference to improve the computing efficiency of the GPU; by dynamically selecting the most suitable computing unit for execution and the cold experts that need to be loaded, efficient resource scheduling is achieved; overall, through the scheme of the application, the communication overhead of the MoE model during expert parallel inference can be significantly reduced, while the computing efficiency of the GPU is improved, and the overall throughput performance during inference is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description only show some of the embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort based on these drawings.

[0014] Figure 1 A schematic diagram of a mixed expert model asynchronous parallel inference system provided by the embodiments of the present application.

[0015] Figure 2 A schematic diagram of the architecture of a mixed expert model asynchronous parallel inference system provided by the embodiments of the present application.

[0016] Figure 3 A schematic diagram of the MoE model calculation process in the existing parallel inference system provided by the embodiments of the present application.

[0017] Figure 4 A schematic diagram of the model calculation process in the asynchronous calculation communication superposition module provided by the embodiments of the present application.

[0018] Figure 5 A schematic diagram of the workflow of the data side loading module in the online stage of the system provided by the embodiments of the present application.

[0019] Figure 6 A schematic diagram of a processing device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.

[0021] Firstly, the terms possibly used in the present text are explained as follows: The terms “include”, “contain”, “have”, “possess” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, the inclusion of a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.) should be interpreted as not only including the explicitly listed technical feature element, but also including other technical feature elements not explicitly listed but known in the art.

[0022] The term "consisting of" denotes an exclusion of any element, not explicitly listed. If the term is used in the claims, the term "consisting of" shall mean an exclusive inclusion, such that no element not specified in the claim shall be read into the claim. If the term is used in the description it shall be taken as equivalent to the term "including".

[0023] Unless specifically stated otherwise, the terms "mounting", "connected", "connecting", "fixed", and "fixing" should be construed as loose terms, for example: can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in this paper can be understood according to the specific circumstances.

[0024] A mixed expert model asynchronous parallel reasoning method, system, device and medium provided by the present application are described in detail below. The contents not described in detail in the embodiments of the present application belong to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conditions are performed according to the conventional conditions in the art or the conditions recommended by the manufacturer. If no manufacturer is specified for the instruments used in the embodiments of the present application, they are all conventional products that can be obtained by market purchase.

[0025] Embodiment one The embodiments of the present application provide a mixed expert model asynchronous parallel reasoning system, which mainly comprises: The asynchronous computing communication superposition module is used for converting full-to-full synchronous communication in expert parallel into point-to-point asynchronous communication, and combining a multi-thread mechanism to realize parallel superposition of mixed expert model computing and word data communication; wherein the mixed expert model computing is controlled by scheduling instructions output by the scheduling execution module.

[0026] The data side loading module is configured to, in an offline stage, pre-run a part of data to be processed, analyze the hot and cold expert distribution in each layer of the hybrid expert model, and analyze the maximum cold expert side loading ratio side loaded to the CPU memory; in an online stage, in each layer of the hybrid expert model, the corresponding hot and cold expert distribution and the maximum cold expert side loading ratio are combined to side load the cold experts to the CPU memory, and the remaining experts are retained in the GPU memory; for the cold experts side loaded to the CPU memory, the cold experts are loaded back to the GPU memory in the granularity of layers under the control of the scheduling instruction for expert calculation; wherein the GPU is a graphics processing unit, the CPU is a central processing unit, the hot and cold expert distribution is measured by the probability of an expert receiving a token, the higher the probability, the higher the possibility of the expert receiving the token, and the lower the probability, the lower the possibility of the expert receiving the token; and the expert calculation is part of the hybrid expert model calculation.

[0027] The scheduling execution module is configured to record the token accumulation in different model layers on the GPU, and generate a scheduling instruction (calculation scheduling) for controlling the hybrid expert model calculation; and record the cold expert side loading, and generate a scheduling instruction (side loading scheduling) for controlling the loading of the cold experts.

[0028] The token accumulation described herein refers to the accumulated number of tokens dynamically collected on the corresponding calculation unit in the inference process, which can be obtained by conventional techniques, and will not be described herein.

[0029] The above method provided by the embodiment of the application mainly has the following advantages: (1) The synchronization of GPU inter-computation and communication caused by All-to-all set communication in the current expert parallelism is decoupled, allowing the MoE model calculation on the GPU and the token data communication to be asynchronously parallel, thereby fully masking the data communication overhead caused by expert parallelism, and eliminating the synchronization waiting overhead.

[0030] (2) According to the phenomenon of uneven distribution of hot and cold experts in the MoE model inference, the hot experts receiving more tokens are placed in the GPU memory, and the cold experts receiving fewer tokens are side loaded in the CPU memory, thereby releasing the GPU memory space, so that the MoE model can improve the GPU calculation efficiency by increasing the processing batch size during inference.

[0031] (3) By dynamically selecting the most suitable calculation unit (including attention units and hybrid expert units, which will be described later) and the cold experts to be loaded, efficient resource scheduling is achieved.

[0032] Overall, by the scheme of the present application, the communication overhead of the MoE model in expert parallel reasoning can be significantly reduced, the computing efficiency of the GPU can be improved, and the overall throughput performance in the reasoning process can be optimized.

[0033] In order to more clearly show the technical solutions provided by the present application and the technical effects generated, the method provided by the embodiments of the present application is described in detail below with specific examples.

[0034] I. Overall introduction of the scheme.

[0035] The embodiments of the present application provide an asynchronous parallel reasoning method for a hybrid expert model, aiming to improve the overall reasoning efficiency of the hybrid expert model using expert parallel reasoning. In order to achieve this goal, the method provided by the present application mainly includes three modules: an asynchronous computing and communication superposition module, a data side loading module, and a scheduling execution module, as shown in Figure 1 .

[0036] In the asynchronous computing and communication superposition module, the synchronization of GPU inter-computing and communication caused by All-to-all set communication in the current expert parallel is decoupled, enabling the model computing and the token data communication on the GPU to be performed asynchronously, thereby reducing the idle time of the GPU, masking the data communication overhead caused by parallel, and eliminating the synchronization waiting overhead. Specifically, the All-to-all set communication of token exchange is decoupled into point-to-point Send-Recv (send-receive) communication that can be performed asynchronously between each GPU, and the implementation of model computing and data communication is split into multiple threads for parallel execution using a multi-thread mechanism, enabling sufficient superposition of computing and data communication.

[0037] In the data side loading module, according to the phenomenon of uneven distribution of hot and cold experts within a layer that commonly exists in MoE model reasoning, the hot experts that receive more tokens are placed in the GPU memory, and the cold experts that receive fewer tokens are side-loaded in the CPU memory, thereby releasing memory space, so that the model can improve the computing efficiency of the GPU by increasing the processing batch size during reasoning. Specifically, during the offline stage before the MoE model performs reasoning, the distribution of hot and cold experts in each layer is obtained by pre-running on a certain scale of data set, and then a part of the cold experts is side-loaded to the CPU memory according to the distribution, releasing the GPU memory space to improve the maximum batch size that can be accommodated by the system during reasoning. For the side-loaded cold experts, they are scheduled back to the GPU for execution according to the accumulation degree of the tokens received during the online stage.

[0038] In the scheduling execution module, scheduling information of token accumulation of each calculation unit in different model layers on the GPU and cold expert side loading is recorded, based on the scheduling information, the calculation unit most suitable for execution and the cold expert to be reloaded are selected under the guidance of the scheduling algorithm, and then the related processes in the above two modules are controlled through scheduling instructions. Specifically, according to the token accumulation of each calculation unit, the scheduling instructions for controlling the mixed expert model calculation in the direction of improving the utilization rate of the GPU are generated based on the calculation resource, the storage resource and the communication resource in the GPU, and are input to the asynchronous calculation communication superposition module; at the same time, the token accumulation progress of the cold expert is monitored, and the cold expert with the highest accumulation degree is selected on the basis of covering the side loading overhead (i.e. the time for loading the cold expert on the GPU is less than the time for the current mixed expert model calculation, and the execution time will not be prolonged due to waiting for the loading of the cold expert), and thus the scheduling instructions for controlling the loading of the corresponding cold expert in the CPU to the GPU are generated and input to the data side loading module.

[0039] Based on the above three modules, not only the communication overhead and waiting overhead caused by the large amount of token synchronous data communication in the MoE model are sufficiently covered through the asynchronous expert parallel, but also the calculation efficiency of the GPU is effectively improved in the form of improving the batch size through the cold expert data side loading, thereby providing a more efficient and economical solution for the parallel inference system of the MoE model.

[0040] II. Detailed introduction of the scheme.

[0041] Figure 2 The detailed architecture of the asynchronous parallel inference method of the mixed expert model provided by the embodiment of the application is shown, and the following will introduce each part in detail.

[0042] 1. Asynchronous calculation communication superposition module.

[0043] In the example of the application, the asynchronous calculation communication superposition module aims to realize asynchronous expert parallel inference, which changes the execution mode of the MoE model inference from model layer granularity, layer-by-layer inference to unit granularity within the model layer, execution according to the scheduling order, allows the requests in one inference batch to have different execution progress, and eliminates the synchronization dependence between the requests for calculation and the requests for token communication; then, through the mechanism of multiple threads, the main thread is responsible for the calculation and execution of the model, and the communication thread is responsible for the communication of token data, so as to enable efficient superposition of model calculation and data communication in the inference process.

[0044] In order to intuitively understand the innovation of the application, the following will introduce the parallel technology used in the parallel inference of the MoE model.

[0045] Figure 3The computing flow of the existing MoE model parallel inference is shown. The MoE model also has the repeated layer structure of the Transformer model, and each layer is mainly composed of an attention layer (Attention layer) for Attention calculation and a mixed expert layer (MoE layer) containing multiple experts and routing. Here, the attention layer and the routing in the mixed expert layer are divided into attention units (Attention unit) responsible for attention calculation and routing; only the part containing expert calculation in the mixed expert layer is divided into a mixed expert unit (MoE unit) responsible for expert calculation. When deploying the MoE model for parallel inference, the current popular solution (such as the open source inference framework SGLang) often adopts different parallel strategies for different units in the model: data parallelism (Data Parallelism, DP) is used for attention units with small parameter proportion, the same parameters are copied to the GPU memory participating in the calculation, and each GPU can independently accept input requests; expert parallelism is used for mixed expert units with large parameter proportion, and the experts in each layer of the mixed expert unit are evenly divided on the GPUs participating in the calculation to reduce the memory pressure.

[0046] When using the above parallel scheme to infer the MoE model, each GPU has a corresponding execution process on the CPU, and the main thread in the execution process will receive the input request, independently perform calculation in the attention unit of each layer, until the Top-K experts to be processed in the mixed expert unit for each token in the request are obtained through the routing result; then, All-to-all synchronization communication will be started on each GPU, and the token data processed by the experts will be sent to the corresponding GPU on other GPU memory, while receiving the token data sent to itself; after all GPUs complete the communication, the GPU will use the experts contained therein to perform expert calculation on the received token; after the expert calculation is completed, the GPU will start All-to-all communication again to restore the tokens contained in the request before entering the mixed expert unit calculation, aggregate the token results calculated on the experts in different GPUs, and then enter the subsequent model layer-by-layer processing until the end of the model execution to produce the output. However, the All-to-all communication of token data between GPUs will cause a large amount of communication overhead, and due to the different hot and cold properties of experts divided on different GPUs, there will be a phenomenon that the GPU with less token reception and short expert calculation time waits for the GPU with more token reception and long expert calculation time to complete the All-to-all restoration token communication before subsequent calculation, further reducing the GPU utilization.

[0047] In order to solve the problem of low GPU utilization in the existing expert parallel based on all-to-all synchronous communication, an asynchronous computing communication superposition module in the expert parallel reasoning method is proposed, as shown in the figure. Figure 4 The asynchronous computing communication superposition module changes the execution mode during model reasoning from model layer granularity, layer-by-layer reasoning to model layer unit granularity, execution according to the scheduling order, allows the requests in one batch to have different execution progress, and creates an opportunity for superposition of model computing and communication overhead. For example, after the attention unit calculation of the first layer is completed, the mixed expert unit of other model layers can be executed according to the scheduling instruction, in this process, the word data communication of the mixed expert unit of the first layer is completed, and then the calculation of the mixed expert unit of the first layer is executed, so that the system does not need to wait for synchronization when performing the word communication of the request of the first layer, and can call the model calculation in other layers to cover the communication overhead.

[0048] Specifically, the asynchronous superposition of model calculation and word data communication in the expert parallel is realized through the mechanism of multi-thread. Each GPU has a corresponding execution process on the CPU, and the execution process includes a main thread and a communication thread; the main thread is a calculation thread, which performs the calculation of the calculation unit of the mixed expert model in different layers by continuously starting the CUDA (Unified Computing Device Architecture) kernel function on the GPU, and the calculation unit includes the attention unit and the mixed expert unit; at the same time, the communication thread on the execution process exchanges the word data according to the routing result obtained by the different attention unit calculations to the GPU where the object expert included in the routing result is located according to the point-to-point asynchronous communication.

[0049] In the embodiment of the application, the GPUs involved are GPUs participating in expert parallel, which will not be described here.

[0050] In the embodiment of the application, the point-to-point asynchronous communication can be a point-to-point notification-transmission two-stage communication, and the exchange of word data is realized through the NCCL Send-Recv (NVIDIA Collective Communication Library, point-to-point send-receive) communication primitive. In this way, the main thread responsible for calculation and the secondary thread responsible for communication can run at the same time, and the superposition of calculation and communication overhead is fully realized.

[0051] ​​​​In the embodiment of the application, the implementation mode of the two-stage point-to-point notification-transmission communication is as follows: each GPU maintains a fixed message queue in the communication thread of the corresponding execution process on the CPU, and the communication thread in the corresponding execution process of each GPU acts as a data receiver or a data sender in the communication process; when the data sender obtains the information of the token routing, the data sender notifies the data receiver of the metadata of the data to be transmitted through the CPU to the message queue of the data receiver, and waits for the data receiver to obtain the metadata from the message queue, which includes information such as the data sender information and the data volume; after the data sender and the data receiver complete the exchange of metadata information, the data sender and the data receiver will respectively start the non-blocking NCCL Send and NCCL Recv communication primitives on the corresponding GPU, and perform the transmission of token data in the background, at the same time, the data receiver and the data sender can also send or receive other token communication tasks without waiting for the current token communication task to end; wherein, NCCL Send is a point-to-point sending of the NVIDIA Collective Communication Library, and NCCL Recv is a point-to-point receiving of the NVIDIA Collective Communication Library.

[0052] 2. Data side loading module.

[0053] In the example of the application, the data side loading module includes: in the offline stage before formal reasoning, the pre-running is used to obtain the layer-by-layer cold-hot expert distribution of the MoE model on the current task and the maximum cold expert side loading ratio that can be masked by the side loading overhead. When entering the online stage, in the model weight loading stage, the coldest expert is side loaded to the CPU layer by layer according to the cold-hot expert distribution and the maximum cold expert side loading ratio obtained in the offline stage until the maximum cold expert side loading ratio is reached, so as to release more video memory space for increasing the batch size during reasoning and improving the calculation utilization rate of the GPU; in the reasoning stage, the system will execute the calculation units in different layers in the asynchronous calculation communication superposition module according to the scheduling, and for the cold experts side loaded in each layer of mixed expert unit, the data side loading module will load them back to the GPU for calculation according to the token accumulation degree in the reasoning process under the condition that the loading overhead is masked.

[0054] In the embodiment of the application, in the offline stage before formal reasoning, the MoE model is pre-run on a small-scale data set, and the cold-hot analyzer and the side loading solver in the data side loading module are used to respectively obtain the cold-hot expert distribution in each layer of the model and the maximum cold expert side loading ratio side loaded to the CPU memory.

[0055] (1) Cold-hot analyzer.

[0056] The cold-hot analyzer inserts a dictionary-like data structure after the routing mechanism of each layer of the hybrid expert model, captures the Top-K experts to which the lexical items are routed in each processing request when a part of the data to be processed is pre-run, and counts the records corresponding to the experts in the dictionary-like data structure; wherein the Top-K experts refer to the top K experts with the highest correlation with the lexical item features. Then, the count results recorded in the dictionary-like data structure of each layer of the hybrid expert model are processed using a normalization index (Softmax) function to convert the absolute number of lexical items received by the experts into the probability of lexical items received by the experts, obtain the probability distribution of lexical items received by the experts, and use the probability distribution of lexical items received by the experts as the cold-hot expert distribution. This probability distribution reflecting the cold-hot of the experts is used as a reference for selecting side-loaded experts in the model weight loading in the online stage.

[0057] (2) Side-loading solver.

[0058] The side-loading solver obtains the changes of available computing resources and communication performance with the amount of data processed by running a series of benchmark tests.

[0059] For data communication between the GPU and the CPU (i.e., loading cold expert data), after setting multiple sizes of data for transmission and sampling the communication delay, a model is used to fit the communication delay performance under different sizes of data, The model is expressed as: wherein, is the time required for transmitting a message with a size of m bytes, is the fixed delay of starting communication, is the data transmission time per byte, which is equivalent to the inverse of the communication bandwidth.

[0060] For matrix multiplication, attention calculation and expert calculation on the GPU, after setting multiple batch sizes (Batch size) of input for calculation delay sampling, a linear fitting form is used to simulate the calculation delay of different types of calculations under different batch sizes between each pair of adjacent batch size sampling points as performance analysis data. Then, combined with the performance analysis data obtained by testing, the relationship between the side-loading proportion of each layer of experts in the hybrid expert model and the maximum batch size is modeled, and the maximum proportion of the loading overhead that can be hidden by the execution time of the single-layer calculation of the hybrid expert model is selected as the maximum cold expert side-loading proportion.

[0061] Specifically: according to the change of the communication delay of the data communication between the GPU and the CPU with the transmission data amount, the loading overhead of reloading the cold experts of one layer to the GPU when the experts are side-loaded at a certain proportion is constructed ​​The calculation delay of each layer of the hybrid expert model combined with the matrix multiplication, attention calculation and expert calculation performed on the GPU varies with the batch size, and the maximum batch size supported by the GPU memory is determined according to the side load setting ratio to construct the execution time of each layer of the model , and the following formula is always guaranteed: ; The maximum cold expert side load ratio is solved under the condition that the formula is established.

[0062] Based on the output of the cold and hot analyzer and the side load solver, the working mode in the online stage is as shown in Figure 5 . The online stage includes a weight loading stage and an inference stage. In the weight loading stage, the parameters in each layer of the hybrid expert model and the attention unit are completely loaded onto all GPU memories participating in inference, and the parameters in the hybrid expert unit (i.e., all experts contained in the hybrid expert unit) are evenly divided into each GPU memory participating in inference according to the number of GPUs, and on each GPU, the cold and hot expert distribution of the experts is determined, and the coldest expert is side-loaded onto the CPU memory until the maximum cold expert side load ratio is reached, and the remaining experts are retained in the GPU memory; then, a data buffer (Buffer) of the size of the side-loaded experts in the layer is opened on the GPU, which is used to store the cold experts loaded back onto the GPU memory for calculation. In the inference stage, the tokens routed to the cold experts during the inference process cannot be calculated immediately and are temporarily stored in the GPU for accumulation of the number of tokens. The data side load module will load the cold experts back onto the GPU memory for expert calculation according to the accumulation of the tokens on the side-loaded cold experts under the control of the scheduling instruction, and the entire loading process will use non-blocking PCIe (high-speed serial computer expansion bus standard) data transmission to ensure that the loading overhead is hidden by the calculation.

[0063] 3. A scheduling execution module.

[0064] In the example of the present application, the scheduling execution module records the scheduling information of the token accumulation and cold expert side loading of each calculation unit in different model layers on the GPU, and based on the scheduling information, the most suitable calculation unit and the cold experts to be reloaded are selected under the guidance of the scheduling algorithm, and then the scheduling instruction is generated and applied to the above two schemes.

[0065] Specifically, each GPU corresponds to a unit scheduler located in the main thread of the execution process on the CPU corresponding to the GPU, uses a dictionary-like data structure to track the location information of each request in the inference batch request, such as the request being located in the attention unit of a specific layer or the accumulation cache of the mixed expert unit of a specific layer, and records the number of cold experts in each layer of the model that are offloaded to the CPU to form scheduling information for reference when scheduling. When the computation on the GPU is completed, the scheduling algorithm in the unit scheduler will refer to the scheduling information to select the appropriate computation unit for the GPU to execute, preventing the GPU from being idle and thus maintaining a high-efficiency computation state. At the same time, the scheduling algorithm will consider the impact of the accumulated word pairs on the available GPU memory of the cold experts that are offloaded, and will load them back to the GPU for computation on the basis of masking the loading overhead of the cold experts.

[0066] The scheduling algorithm will preferentially schedule the mixed expert unit with a larger number of accumulated word pairs for execution, and secondarily schedule the attention unit containing a larger number of word pairs for execution, so that the units containing a large batch are calculated first to ensure the utilization rate of the GPU; if the number of word pairs blocked by the cold experts is too large, the word pairs occupy a large KV (key-value data) cache, exceeding the threshold value that affects the normal operation of the system, then the cold experts of the model layer with the largest number of accumulated word pairs are loaded back to the GPU for calculation, and then the calculation is scheduled and the blocked word pairs are released to the execution of the subsequent unit to promote the early release of the KV cache.

[0067] The above scheme provided by the embodiments of the present application can deploy each module on the CPU in the implementation process, and in the inference process, the CPU controls each module to submit calculation, communication, or cold expert loading tasks to the GPU.

[0068] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above.

[0069] Embodiment two The present application also provides an asynchronous parallel inference method for a mixed expert model, which is mainly based on the system provided in the foregoing embodiments. The method mainly includes: Converting the all-to-all synchronous communication in expert parallelism into point-to-point asynchronous communication, and combining a multi-thread mechanism to realize the parallel superposition of mixed expert model calculation and word data communication; wherein the mixed expert model calculation is controlled by the scheduling instruction output by the scheduling execution module; In the offline phase, a portion of the data to be processed is pre-run to analyze the distribution of hot and cold experts in each layer of the hybrid expert model, as well as the maximum proportion of cold experts loaded onto CPU memory. In the online phase, for each layer of the hybrid expert model, based on the corresponding distribution of hot and cold experts and the maximum proportion of cold experts loaded onto CPU memory, cold experts are loaded onto CPU memory, while the remaining experts are retained in GPU memory. For cold experts loaded onto CPU memory, under the control of scheduling instructions, they are loaded back into GPU memory at the layer level for expert computation. Here, GPU is the graphics processing unit, and CPU is the central processing unit. The distribution of hot and cold experts is measured by the probability that an expert receives a word. Experts with higher probabilities are more likely to receive words and are called hot experts, while experts with lower probabilities are less likely to receive words and are called cold experts. The expert computation is part of the hybrid expert model computation. Record the lexical accumulation in different model layers on the GPU and generate scheduling instructions for controlling the computation of hybrid expert models; and record the loading of cold experts and generate scheduling instructions for controlling the loading of cold experts.

[0070] Since the main technical details involved in this method have been described in detail in the previous embodiments, they will not be repeated here.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0072] Example 3 The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0073] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0074] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: The input device can be a touch screen, an image acquisition device, a physical button or a mouse, etc. The output device can be a display terminal. The memory can be a random access memory (RAM) or a non-volatile memory such as a disk memory.

[0075] Embodiment Four The application further provides a readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the foregoing embodiments.

[0076] The readable storage medium in the embodiments of the application as a computer readable storage medium can be arranged in the foregoing processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk or various media capable of storing program codes.

[0077] The above merely describes the preferred embodiments of the application, but the protection scope of the application is not limited to this. Any changes or replacements within the technical scope disclosed by the application can be easily conceived by those skilled in the art, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims. The information disclosed in the background section of this document is merely intended to deepen the understanding of the general background of the application and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known by those skilled in the art.

Claims

1. An asynchronous parallel inference system for hybrid expert models, characterized in that, The application relates to a parallel computing communication superposition module for converting full-to-full synchronous communication in an expert parallel into point-to-point asynchronous communication and realizing parallel superposition of mixed expert model computing and word data communication in combination with a multi-thread mechanism; wherein the mixed expert model computing is controlled by scheduling instructions output by a scheduling execution module. A data side loading module is used to pre-run a part of to-be-processed data in an offline stage, analyze cold and hot expert distribution in each layer of the mixed expert model, and analyze the maximum cold expert side loading proportion side loaded to CPU memory; in an online stage, cold experts are side loaded to CPU memory in combination with the corresponding cold and hot expert distribution and the maximum cold expert side loading proportion in each layer of the mixed expert model, and the remaining experts are reserved in GPU memory; under the control of the scheduling instructions, the cold experts side loaded to the CPU memory are loaded back to the GPU memory in a layer granularity for expert computing; wherein the GPU is a graphics processing unit, the CPU is a central processing unit, the cold and hot expert distribution is measured by the probability of expert receiving words, the higher the probability, the higher the possibility of the expert receiving words, and the lower the probability, the lower the possibility of the expert receiving words; the expert computing belongs to part of the mixed expert model computing. A scheduling execution module is used to record word accumulation in different model layers on the GPU, generate scheduling instructions for controlling the mixed expert model computing, and record cold expert side loading, and generate scheduling instructions for controlling the loading of the cold experts. The parallel superposition of the mixed expert model computing and the word data communication in combination with the multi-thread mechanism comprises the following steps:

2. The hybrid expert model's asynchronous parallel inference system of claim 1, wherein, Each layer of the mixed expert model comprises an attention unit and a mixed expert unit, the attention unit is responsible for attention computing and routing, and the mixed expert unit is responsible for expert computing; Each GPU has a corresponding execution process on the CPU, and the execution process comprises a main thread and a communication thread; the main thread is a computing thread, and the computing of the computing units in different layers of the mixed expert model is realized by continuously starting CUDA kernel functions on the GPU; wherein CUDA is a unified computing device architecture, and the computing units comprise the attention unit and the mixed expert unit; Meanwhile, the communication thread on the execution process exchanges word data according to the routing results obtained by different attention units, and the word data is exchanged according to point-to-point asynchronous communication. The point-to-point asynchronous communication is point-to-point notification-transmission two-stage communication, and the implementation manner is as follows:

3. The hybrid expert model's asynchronous parallel inference system of claim 2, wherein, Each GPU maintains a fixed message queue in the communication thread of the corresponding execution process on the CPU, and the communication thread in the corresponding execution process of each GPU acts as a data receiver or a data sender in the communication process; when the data sender obtains the information of the word routing, the data sender notifies the data receiver of the metadata of the to-be-transmitted data through the CPU, and waits for the data receiver to obtain the metadata from the message queue, wherein the metadata comprises information of the data sender and data volume. ​ When the data sender and the data receiver complete the exchange of the metadata information, the data sender and the data receiver start non-blocking NCCL Send and NCCL Recv communication primitives on the corresponding GPUs to transmit the token metadata, and meanwhile, the data receiver and the data sender can also send or receive other token communication tasks; wherein, NCCL Send is a point-to-point sending of the NVIDIA Collective Communication Library, and NCCL Recv is a point-to-point receiving of the NVIDIA Collective Communication Library.

4. The hybrid expert model's asynchronous parallel inference system of claim 1, wherein, The analysis of the hot and cold expert distribution in each layer of the hybrid expert model comprises: A dictionary-like data structure is inserted after the routing mechanism of each layer of the hybrid expert model, and when a part of the data to be processed is pre-run, the Top-K experts to which the token is routed in each processing request are captured, and the counting is performed at the record corresponding to the experts in the dictionary-like data structure; wherein, the Top-K experts refer to the top K experts with the highest correlation with the token features; The counting results recorded in the dictionary-like data structure of each layer of the hybrid expert model are processed using a normalized exponential function to convert the absolute number of tokens received by the experts into the probability of tokens received by the experts, and the probability distribution of tokens received by the experts is obtained, which is used as the hot and cold expert distribution.

5. The hybrid expert model's asynchronous parallel inference system of claim 1, wherein, The analysis of the maximum cold expert side loading ratio loaded to the CPU memory comprises: A series of benchmark tests are run to obtain the available computing resources and communication performance as a function of the amount of data processed; for the data communication between the GPU and the CPU, after setting a plurality of sizes of the amount of data to be communicated and sampling the communication delay, a model is used to fit the communication delay performance under different sizes of the amount of data The model is expressed as: The model is expressed as: ； wherein, time needed to transmit a message of size m bytes, fixed delay for the start of the communication, time for transmission of one byte of data, equivalent to the inverse of the communication bandwidth; For matrix multiplication, attention calculation and expert calculation on the GPU, multiple batch sizes are set for input to calculate the delay sampling, and then a linear fitting form is used between each pair of adjacent batch size sampling points to simulate the calculation delay of different types of calculations under different batch sizes as performance analysis data; Subsequently, combined with the performance analysis data obtained by testing, the relationship between the side load ratio of each layer of experts in the mixed expert model and the maximum batch size is modeled, and the maximum ratio of loading overhead that can be hidden by the single-layer calculation execution time of the mixed expert model is selected as the maximum cold expert side load ratio, including: according to the change of the communication delay of the data communication between the GPU and the CPU with the transmission data volume, the loading overhead of the cold expert of one layer when the side load setting ratio expert is constructed , combined with the change of the calculation delay of each layer of the mixed expert model on the GPU with the batch size, according to the maximum batch size supported by the GPU memory when the side load setting ratio cold expert, the execution time of each layer of the model is constructed , the following formula is always true: ； And the maximum cold expert side loading ratio is solved under the condition that the formula is established.

6. The hybrid expert model's asynchronous parallel inference system of claim 1, wherein, In the online phase, in each layer of the hybrid expert model, the corresponding hot and cold expert distribution and the maximum cold expert side loading ratio are combined to side load the cold experts to the CPU memory, and the remaining experts are retained in the GPU memory; For the cold experts side loaded to the CPU memory, under the control of the scheduling instruction, the cold experts are loaded back to the GPU memory for expert calculation in a layer granularity, which comprises: The online phase comprises a weight loading phase and an inference phase; In the weight loading phase, each layer of the hybrid expert model comprises an attention unit and a hybrid expert unit, the attention unit is responsible for attention calculation and routing, and the hybrid expert unit is responsible for expert calculation, and the parameters in the attention unit are completely loaded to all GPU memories participating in inference; the parameters in the hybrid expert unit are all experts contained in the hybrid expert unit, which are evenly divided into each GPU memory participating in inference according to the number of GPUs, and on each GPU, the cold experts are side loaded to the CPU memory from the coldest expert according to the hot and cold expert distribution of the experts, until the maximum cold expert side loading ratio is reached, and the remaining experts are retained in the GPU memory; thereafter, a data buffer with the size of the side loaded experts in the layer is opened on the GPU, which is used to store the cold experts loaded back to the GPU for calculation. In the reasoning phase, the tokens routed to the cold experts in the reasoning process will be temporarily stored in the GPU for accumulation of the number of tokens, and the data side-loading module will load the cold experts into the GPU memory for expert calculation according to the accumulation of the tokens on the side-loaded cold experts and under the control of the scheduling instruction.

7. The hybrid expert model's asynchronous parallel inference system of claim 1, wherein, The accumulation of the tokens in different model layers on the GPU is recorded, and a scheduling instruction for controlling the mixed expert model calculation is generated; And the side-loading of the cold experts is recorded, and a scheduling instruction for controlling the loading of the cold experts is generated. According to the accumulation of the tokens in each layer of the calculation unit, the scheduling instruction for controlling the mixed expert model calculation is generated in the direction of improving the utilization rate of the GPU, and is input into the asynchronous calculation communication superposition module; wherein each layer of the mixed expert model includes two types of calculation units, namely attention units and mixed expert units, the attention units are responsible for attention calculation and routing, and the mixed expert units are responsible for expert calculation; The accumulation progress of the tokens of the cold experts is monitored and recorded, and the cold expert with the highest accumulation degree is selected on the basis of covering the side-loading overhead, so as to generate a scheduling instruction for controlling the loading of the corresponding cold expert from the CPU to the GPU memory, and input it into the data side-loading module; wherein covering the side-loading overhead means that the time of loading the cold expert from the CPU to the GPU is less than the time of the current mixed expert model calculation.

8. An asynchronous parallel inference method of hybrid expert models, characterized in that, The system implementation based on any one of claims 1-7 comprises: The full-to-full synchronous communication in the expert parallel is converted into point-to-point asynchronous communication, and the multi-threading mechanism is combined to realize the parallel superposition of the mixed expert model calculation and the token data communication; wherein the mixed expert model calculation is controlled by the scheduling instruction output by the scheduling execution module; In the offline phase, a part of the data to be processed is pre-run, the distribution of the cold and hot experts in each layer of the mixed expert model is analyzed, and the maximum cold expert side-loading ratio side-loaded to the CPU memory is analyzed; in the online phase, in each layer of the mixed expert model, the corresponding cold and hot expert distribution and the maximum cold expert side-loading ratio are combined to side-load the cold experts to the CPU memory, and the remaining experts are retained in the GPU memory; for the cold experts side-loaded to the CPU memory, they are loaded back to the GPU memory for expert calculation under the control of the scheduling instruction; wherein the GPU is a graphics processing unit, the CPU is a central processing unit, the cold and hot expert distribution is measured by the probability of receiving tokens, the higher the probability, the higher the possibility of receiving tokens, and the lower the probability, the lower the possibility of receiving tokens, which is called a cold expert; the expert calculation is part of the mixed expert model calculation; The accumulation of the tokens in different model layers on the GPU is recorded, and a scheduling instruction for controlling the mixed expert model calculation is generated; and the side-loading of the cold experts is recorded, and a scheduling instruction for controlling the loading of the cold experts is generated.

9. A processing device, characterized by Comprise: One or more processors; Memory for storing one or more programs; wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement a method as claimed in claim 8.

10. A readable storage medium, storing a computer program, characterized in that, A computer program that when executed by a processor implements a method as claimed in claim 8.

Images