Efficient reasoning method for hybrid expert model

Abstract

According to the efficient reasoning method of the hybrid expert model, calculation of a gating network is integrated into an assembly line, an assembly line strategy is optimized, and a data synchronization mechanism and a routing algorithm are improved, so that efficient parallelism of calculation and communication tasks is realized, communication bandwidth waste is reduced, extra communication overhead is reduced, and communication efficiency is improved. Therefore, the deployment efficiency and performance of the large-scale language model are improved, and the problems that in the prior art, calculation and communication tasks are not fully overlapped, communication bandwidth is wasted, and communication overhead is extra are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of large-model reasoning and computing, and in particular to an efficient reasoning method for a hybrid expert model. Background Art

[0002] In recent years, with the introduction of ChatGPT, large language models (LLMs) have demonstrated powerful capabilities in the field of natural language processing (NLP), driving significant progress in their development and application. From the early BERT model (with 100 million parameters) to the latest Llama 3 model (with 400 billion parameters), the complexity of LLMs has increased significantly. However, the rapid growth in model size poses severe challenges to device memory and the processing power of hardware accelerators. To address these challenges, researchers have proposed a variety of innovative computing paradigms aimed at improving the feasibility of LLM deployment.

[0003] Among these innovations, the Mixture-of-Experts (MoE) model stands out as a key approach. Unlike traditional optimization techniques such as pruning, sparse computing, and quantization, MoE not only improves computational efficiency but also enhances the generalization capabilities of LLMs. Research has shown that a 1.3 billion parameter MoE model containing 128 experts can perform comparable to a dense model with 6.7 billion parameters, significantly reducing model size while maintaining model accuracy. These advantages make MoE the preferred solution for model expansion and have been widely used in multiple fields.

[0004] Furthermore, the inherent nature of MoEs—each expert performs independent computations—has given rise to a new parallelization strategy, known as expert parallelism. This strategy distributes experts across different devices, significantly improving system efficiency and becoming a mainstream strategy for large-scale LLM deployments. However, expert parallelism requires synchronizing input data between devices and transmitting results back to the original device. These synchronization operations inevitably block subsequent computational tasks, reducing the efficiency of MoE execution.

[0005] To address these efficiency issues, recent research works attempt to overlap the execution delay during data transfer with ongoing computation tasks by integrating pipelining strategies into MoE, such as Figure 1 For example, ColossalAI uses the OpenMoE approach to split the transmission of input data into multiple parts, thereby masking the communication overhead of the latter part with the computation of the former. Furthermore, FasterMoE splits the synchronization process at the communication group granularity, enabling experts to process data from a specific device while receiving data from other devices.

[0006] However, existing MoE pipeline implementations remain imperfect, failing to fully overlap computation and communication tasks. For example, gated computations have not yet been integrated into the pipeline, impacting the overall efficiency of the system. Furthermore, current approaches often pad intermediate activations to a fixed length to facilitate synchronization, resulting in wasted communication bandwidth on transmitting padding values. Furthermore, the routing mechanisms in these systems operate at the granularity of experts, introducing additional communication overhead.

[0007] In summary, although MoE and its parallel strategy have made significant progress in improving the efficiency of LLMs, there are still many technical bottlenecks that urgently need further optimization and innovation. Summary of the Invention

[0008] The purpose of the technical solution of the present invention is to integrate the gated network of hybrid expert model reasoning into the pipeline, aiming to solve the problems of insufficient overlap of computing and communication tasks, waste of communication bandwidth and additional communication overhead in the existing technology.

[0009] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention provides an efficient reasoning method of a hybrid expert model, comprising the following steps:

[0010] Group experts on the same device into the same expert group;

[0011] The tokens of the input data are divided into multiple micro-batches. The gated computation is decomposed into multiple subtasks based on the multiple micro-batches. The gated computation and communication tasks between different micro-batches are executed in an overlapping manner.

[0012] Collect metadata about the number of tokens assigned to each expert and exchange the metadata between devices. Perform a single communication on the same expert group to obtain the data required by the single group of experts. Dynamically adjust the amount of data transmitted based on the metadata.

[0013] When the results are aggregated, a single aggregation communication operation is performed on the data within the same expert group.

[0014] Preferably, the gating calculation of each micro-batch is performed in parallel with the data distribution task of the previous micro-batch.

[0015] Preferably, the steps of decomposing into multiple subtasks are as follows:

[0016] Input a single mini-batch of size N×D model Data with shape D model ×N expert The weight matrix is ​​multiplied to generate the output matrix, where N is the number of tokens, D model Embedding dimensions for the model;

[0017] Each element of the output matrix O i,jIt represents the probability of assigning the i-th token to the j-th expert. The topk function selects the k with the highest probability in each row to generate multiple subtasks.

[0018] The technical solution of the present invention proposes an efficient reasoning method for hybrid expert models. By integrating the calculation of the gated network into the pipeline, optimizing the pipeline strategy, improving the data synchronization mechanism and routing algorithm, efficient parallelization of computing and communication tasks is achieved, reducing the waste of communication bandwidth and lowering the additional communication overhead, thereby improving the deployment efficiency and performance of large-scale language models, and solving the problems of insufficient overlap of computing and communication tasks, waste of communication bandwidth, and additional communication overhead in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the inference scheme of the existing hybrid expert model;

[0020] Figure 2 A schematic diagram of the pipeline execution of the hybrid expert provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0022] An embodiment of the present invention provides an efficient reasoning method for a hybrid expert model, comprising the following steps:

[0023] Expert grouping during the model deployment phase: Experts are grouped based on the location of their devices. Experts on the same device are grouped together, and input data is routed by group, providing guidance for communication aggregation.

[0024] During the model inference phase, token batching is used to divide the input data into multiple micro-batches. Token batching: Since the computation of each token is independent, the input can be partitioned along N dimensions to generate multiple micro-batches. The gating computation of each micro-batch can be performed in parallel with the data scattering task of the previous micro-batch, thereby reducing the total execution time of the hybrid expert module.

[0025] The gated computation is decomposed into multiple subtasks based on token partitioning, and the subtasks are overlapped with the communication tasks to improve the efficiency of the MoE pipeline execution. The specific steps include:

[0026] Decomposition of gated computation: For input size N×Dmodel Data (where N is the number of tokens, D model is the model embedding dimension), the gate calculation is performed by combining the input with the shape of D model ×N expert The weight matrix is ​​multiplied to generate the output matrix. Each element O of the output matrix i,j represents the probability of assigning the i-th token to the j-th expert. Subsequently, the topk function selects the k highest probability assignments in each row.

[0027] The pipeline execution process of the hybrid expert model has more covered stages, such as Figure 2 shown. Figure 2 The tasks of the same color represent the five atomic task units divided by the MoE reasoning process: gated computing (G), data distribution (S), expert computing (E), result aggregation (G), and combined output (C). The following numbers represent the micro-batch number to which the token belongs.

[0028] In the inference data distribution phase, metadata about the number of tokens assigned to each expert is first collected, and the parallel execution of gating computation and data distribution tasks is initiated.

[0029] Adaptive on-demand data transmission based on metadata, dynamically adjusting data transmission volume and memory allocation.

[0030] Metadata Collection: Metadata about the number of tokens assigned to each expert is first collected and this information is exchanged between devices to dynamically allocate storage space and optimize data transmission.

[0031] After obtaining the specific amount of transmitted data, according to the expert grouping in the model deployment phase, the communication of expert data within the same group is aggregated, and a single communication is used to obtain the data required by a single group of experts.

[0032] On-demand transmission: Dynamically adjust the amount of data transmitted based on metadata to avoid transmitting invalid padding data, thereby reducing the waste of communication bandwidth.

[0033] In the result aggregation stage of inference, consistent with the communication method in the data distribution stage, aggregate communication is used for data transmission to improve the utilization of memory bandwidth while reducing communication latency. Packet routing based on transmission aggregation is used to reduce the communication overhead caused by the increase in the number of experts.

[0034] Transfer aggregation: Perform a single aggregate communication operation on expert data within the same group, reducing protocol-related setup costs and synchronization overhead while improving memory bandwidth utilization.

[0035] Communication frequency optimization: By reducing the communication frequency, the scalability of the system is improved, especially when there are a large number of experts, the communication delay is significantly reduced.

[0036] By integrating gated computation into the pipeline and overlapping it with communication tasks, the embodiments of the present invention achieve efficient parallel execution of computation and communication tasks, significantly improving computational resource utilization. Simultaneously, through an adaptive on-demand transmission strategy, the transmission of invalid data is reduced, optimizing communication bandwidth utilization. Furthermore, a design based on packet routing and transmission aggregation reduces communication frequency and synchronization latency, improving system scalability and throughput. These technical improvements collectively address existing hybrid expert model reasoning issues, such as insufficient overlap between computation and communication tasks, wasted communication bandwidth, and excessive communication overhead. Ultimately, this enables efficient hybrid expert model reasoning, significantly improving the deployment efficiency and performance of large-scale language models.

Claims

1. An efficient reasoning method for a hybrid expert model, characterized in that: The following steps are involved: Group experts on the same device into the same expert group; The tokens of the input data are divided into multiple micro-batches. The gated computation is decomposed into multiple subtasks based on the multiple micro-batches. The gated computation and communication tasks between different micro-batches are executed in an overlapping manner. Collect metadata about the number of tokens assigned to each expert and exchange the metadata between devices. Perform a single communication on the same expert group to obtain the data required by the single group of experts. Dynamically adjust the amount of data transmitted based on the metadata. When the results are aggregated, a single aggregation communication operation is performed on the data within the same expert group.

2. The efficient reasoning method of a hybrid expert model according to claim 1, characterized in that: The gating calculation of each micro-batch is performed in parallel with the data distribution task of the previous micro-batch.

3. The efficient reasoning method of a hybrid expert model according to claim 1, characterized in that: The steps of decomposing into multiple subtasks are as follows: Input a single mini-batch of size N×D model Data with shape D model ×N expert The weight matrix is ​​multiplied to generate the output matrix, where N is the number of tokens, D model Embedding dimensions for the model; Each element of the output matrix O i,j It represents the probability of assigning the i-th token to the j-th expert. The topk function selects the k with the highest probability in each row to generate multiple subtasks.

Images