Assembly line task arrangement method in hybrid expert model reasoning
By decomposing tasks and constructing constraint rules for the hybrid expert model, tasks are ensured to be executed in a preset order, solving the mismatch problem between pipeline orchestration and hardware scheduling when hardware resources are limited or computing and communication tasks are intertwined, thereby improving system efficiency.
Patent Information
- Application Number
- CN202510663815.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-22
- Publication Date
- 2025-09-23
AI Technical Summary
In existing technologies, when the hardware resources of the hybrid expert model are limited or the computing and communication tasks are intertwined, the pipeline orchestration and hardware scheduling are mismatched, resulting in a degradation of the overall system performance.
By decomposing the reasoning process of the hybrid expert model into four task units: gated calculation, input distribution, expert calculation, result aggregation, and combined output, and constructing order constraints within micro-batches, order constraints for homogeneous tasks across micro-batches, global constraints for communication tasks, and metadata-data dependency constraints, the GPU stream and event synchronization mechanism are utilized to ensure that tasks are executed in the preset order.
It improves hardware resource utilization, enhances the overall operating efficiency of the system, solves the mismatch between pipeline orchestration and hardware scheduling, and achieves more efficient hybrid expert model parallel reasoning.
Smart Images
Figure CN120687208A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of large model reasoning and computing, and in particular to a pipeline task scheduling method in hybrid expert model reasoning. Background Art
[0002] Parameters have evolved to reach hundreds of billions in models like Llama 3. While this trend improves model performance, it also places stringent demands on hardware deployment: the memory capacity of a single device is insufficient to support the entire model, and the computational throughput of hardware accelerators struggles to keep up with the demands of intensive computing. Traditional lightweighting techniques such as model pruning, quantization, and sparse computing can reduce resource consumption, but often come at the expense of model expressiveness, making it difficult to achieve both efficiency and performance.
[0003] In this context, the Mixture-of-Experts (MoE) model has become a breakthrough solution due to its dynamic routing mechanism. This architecture splits the model into multiple independent expert modules and dynamically activates some experts based on input features, significantly reducing the number of parameters (for example, a 130 million-parameter MoE model can match the performance of a 670 million-parameter dense model) while retaining the model's generalization capabilities. However, the efficient deployment of MoEs relies heavily on expert parallelization strategies, which involves collaboratively executing expert computations across distributed devices. This process faces a core bottleneck of deep coupling between communication and computation.
[0004] Since experts need to frequently synchronize input data and calculation results between devices during parallel processing, their communication delay accumulates linearly with the number of experts. In addition, the dynamic routing mechanism leads to unbalanced loads between devices, causing alternating idle waits and overload blocking. More importantly, the multi-level task queue mechanism of hardware platforms such as GPUs is difficult to adapt to the high concurrency and fine-grained computing characteristics of MoE, resulting in unpredictable deviations in the order of task execution due to resource contention. Existing optimization schemes attempt to mask communication overhead through pipeline strategies, such as transmitting input data in segments in ColossalAI to overlap communication and calculation, or splitting the synchronization process by device group granularity in FasterMoE to achieve local parallelism.
[0005] However, existing methods of this type do not fully consider the mismatch between the actual scheduling behavior and the original pipeline layout when hardware resources are limited or computing and communication tasks are interleaved. This will cause the preset pipeline order to deviate from the actual execution process, reduce the task overlap rate of the pipeline, and ultimately lead to a decline in overall system performance. Summary of the Invention
[0006] The technical problem to be solved by the technical solution of the present invention is: to solve the problem of mismatch between pipeline orchestration and hardware scheduling in parallel reasoning of hybrid expert models when hardware resources are limited or computing and communication tasks are intertwined, thereby improving hardware resource utilization and enhancing the overall operating efficiency of the system.
[0007] In order to solve the above technical problems, the technical solution of the present invention provides a pipeline task scheduling method in hybrid expert model reasoning, comprising the following steps:
[0008] The reasoning process of the hybrid expert model is decomposed into tasks, and the gated calculation G, expert calculation E, combined output C for computationally intensive tasks, and input distribution S and result aggregation R for cross-device communication operations are obtained. The input distribution S is further split into metadata transmission S. M With data block transfer S D .
[0009] Construct the order constraint C1 within the micro-batch according to the order of gate calculation G, input distribution S, expert calculation E, result aggregation R, and combined output C.
[0010] The gated computation G includes the first gated computation task G i and the second gated computation task G i+1 , according to the first gated computing task G i and the second gated computation task G i+1 In , the order of micro-batches between the i-th micro-batch and the i+1-th micro-batch is used to construct the order constraint C2 of isomorphic tasks across micro-batches.
[0011] For n micro-batch gated computing tasks (G1, G2…, G n ), with n input radiating (S1, S2…, S n ), according to the n input distribution (S1, S2…, S n ) and then perform result aggregation R to build the global constraint C3 of the communication task.
[0012] According to the metadata transmission within the same micro-batch i, Prioritize block transfers When crossing micro-batches i and i+1, the i-th micro-batch data block is transferred Prioritizes metadata transmission over the i+1th micro-batch Completed before, build metadata-data dependency constraint C4.
[0013] For n micro-batch gated computations G and expert computations E, we use the cross-micro-batch homogeneous task order constraint C2 and utilize the sequential execution characteristics within the stream to allocate them to independent GPU streams for execution. Specifically, we put the same computation tasks into the same stream.
[0014] For cross-stream dependent tasks that follow the order constraint C1 within the micro-batch, the global constraint C3 for communication tasks, and the metadata-data dependency constraint C4, event synchronization is established in different stream tasks to enforce the execution order of different stream tasks and build inter-stream constraint implementation. Specifically, the tasks executed later in the same micro-batch use events to depend on the tasks executed earlier (C1), all result aggregation tasks use events to depend on the input distribution tasks (C3), and the metadata data transmission of the i+1th micro-batch is Using events depends on the i-th micro-batch data block transmission (C4).
[0015] During the task submission phase, the reasoning process of the hybrid expert model dynamically decides the task submission timing and performs checks based on the intra-micro-batch order constraint C1, the cross-micro-batch isomorphic task order constraint C2, the global constraint C3 of the communication task, the metadata-data dependency constraint C4, the intra-stream constraint implementation, and the inter-stream constraint implementation.
[0016] Preferably, the i-th micro-batch gating calculation G in the n micro-batch gating calculations G is i Submit to StreamG to ensure that the gate calculation G of the i-th micro-batch i After completion, the i+1th micro-batch gate calculation G is automatically triggered i+1 .
[0017] Preferably, the different stream tasks are the i-th micro-batch inputs that belong to different streams and distribute S i and the i-th mini-batch expert calculates E i , S is distributed in the i-th micro-batch input i Record the event cudaEventRecord(event Si ,stream S ), in the i-th micro-batch expert calculates E i Insert event before submitting wait cudaStreamWaitEvent(stream E ,event Si ), to ensure that the i-th micro-batch expert calculates E i S is distributed only on the i-th micro-batch input i After submission, it is started to implement the aforementioned mandatory constraints on the execution order of different flow tasks.
[0018] Preferably, the checking includes: when the i-th mini-batch expert calculates E i When the software layer is ready, verify that the i-th micro-batch input emits S iWhether the event completion mark in the flow or the implicit synchronization signal of the communication interface has been met, if so, the task is submitted; if not, the submission is delayed until the dependency is released.
[0019] The technical solution of the present invention proposes a pipeline task scheduling method in hybrid expert model reasoning. By establishing multi-level constraint rules, reconstructing the adaptation logic of the GPU task submission mechanism and the synchronization interface, the submission order of tasks in the GPU stream, event and communication interface is precisely controlled to ensure that the preset pipeline arrangement is consistent with the actual hardware execution process, and solving the efficiency loss problem caused by the mismatch between the existing static pipeline scheduling and the dynamic behavior of hardware scheduling. Furthermore, it solves the problem of mismatch between pipeline scheduling and hardware scheduling in hybrid expert model parallel reasoning when hardware resources are limited or computing and communication tasks are intertwined, thereby improving hardware resource utilization and enhancing the overall operating efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 The actual execution order provided by the embodiment of the present invention conforms to the preset pipeline order;
[0021] Figure 2 This is a dependency modeling solution for orchestrating tasks in a hybrid expert model reasoning pipeline according to an embodiment of the present invention. DETAILED DESCRIPTION
[0022] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.
[0023] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a pipeline task scheduling method in hybrid expert model reasoning, comprising the following steps:
[0024] The MoE reasoning process is divided into five atomic task units for task decomposition: gated computing (G), input distribution (S), expert computing (E), result aggregation (R), and combined output (C). Among them, G, E, and C are computationally intensive tasks, S and R involve cross-device communication operations, and the input distribution task is further divided into metadata transmission (S M ) and data block transfer (S D Based on this division, the following four-level constraint rules are constructed to define the dependencies between tasks for dependency modeling:
[0025] 1. Order constraint within micro-batch (C1): The execution of tasks in a single micro-batch strictly follows G i →Si →E i →R i →C i The order ensures that gating decisions take precedence over data distribution, and aggregation operations are triggered after the expert calculation results are complete.
[0026] 2. Cross-micro-batch homogeneous task order constraint (C2): The same type of tasks are strictly executed in sequence across different micro-batches (e.g. G i →G i+1 ), so that the computation tasks of the i-th micro-batch can overlap with the communication tasks of the i+1-th micro-batch while avoiding task reordering conflicts.
[0027] 3. Communication task global constraint (C3): All input distribution tasks (S1, S2, ..., S n ) must be executed in any result aggregation task (R j ) before startup, eliminating the risk of disorder in communication scheduling and preventing subsequent computing tasks from being blocked due to unready data.
[0028] 4. Metadata-data dependency constraint (C4): Within the same micro-batch, metadata data is transmitted Prioritize block transfers Ensure that the receiving end parses the data block according to the metadata; when crossing micro batches, $S_i^D$ needs to be It is completed before the data pipeline is completed, preventing deadlock or scheduling uncertainty caused by the interleaving of metadata and data pipelines.
[0029] Based on the characteristics of the GPU hardware abstraction layer, four types of constraint rules are implemented in a differentiated manner to build a constraint implementation mechanism, including:
[0030] 1. Implementation of intra-stream constraints in C2 (stream serial execution of tasks): Assign tasks of the same type (e.g., all G tasks, all E tasks) to the stream scheduling hardware unit in the independent NVIDIA GPU stream for execution, and use the sequential execution characteristics of the stream to ensure that homogeneous tasks across micro-batches are submitted strictly in sequence. For example, all G i All tasks are submitted to Stream G Flow, ensure G i Automatically trigger G after completion i+1 , no explicit event synchronization is required, thus minimizing the intra-stream scheduling overhead.
[0031] 2. Implementation of inter-stream constraints for C1 / C3 / C4 (inter-stream task synchronization): For tasks involving cross-stream dependencies (such as S i With E iBelong to different streams), and enforce the execution order through event synchronization mechanisms, such as through NVIDIA GPU's cudaevent related instructions (such as cudaEventWait), as follows:
[0032] a) In S i Record events when tasks are submitted (cudaEventRecord(event Si ,stream S ));
[0033] b) In E i Insert event wait (cudaStreamWaitEvent(stream E ,event Si )), ensure E i Only in S i Launch it after completion.
[0034] In the task submission phase, the reasoning process checks the task dependencies defined above and dynamically decides the task submission timing according to the above constraint rules. i ) When the software layer is ready, the reasoning framework verifies its predecessor tasks (S i ) Whether the event completion marker in the flow or the implicit synchronization signal of the communication interface has been satisfied. If not, the submission is delayed until the dependency is released to perform dynamic constraint-driven task submission. Specifically:
[0035] 1. In-stream order autonomy: for homogeneous tasks constrained by C2 (such as all G i ), directly leveraging the sequential execution characteristics of streams to ensure order across micro-batches without the need for additional synchronization operations.
[0036] 2. Inter-stream event synchronization: For cross-stream tasks with C1 / C3 / C4 constraints, event recording and wait operations are used to enforce synchronization, ensuring that tasks are executed in the preset order even in the presence of resource contention or scheduler non-deterministic behavior.
[0037] 3. Making implicit communication dependencies explicit: To avoid implicit order dependencies introduced by communication processes (such as ncclAllToAll) in the pipeline, C3 constraints implement dependencies between communication tasks that may reduce pipeline efficiency as explicit event dependency rules, preventing pipeline interruptions caused by disordered communication tasks.
[0038] This embodiment of the present invention provides a pipelined task orchestration method for hybrid expert model reasoning, suitable for supporting streaming serial task execution, inter-stream task synchronization, large language models across all hybrid expert computation layers, and all computing devices involved in device-to-device communication. This embodiment of the present invention is also applicable to conventional server scenarios, where the host-side processor is primarily a CPU, and the device-side processor can be the computing device mentioned above.
Claims
1. A pipeline task scheduling method in hybrid expert model reasoning, characterized in that: The following steps are involved: The reasoning process of the hybrid expert model is decomposed into tasks, and the gated calculation G, expert calculation E, combined output C for computationally intensive tasks, and input distribution S and result aggregation R for cross-device communication operations are obtained. The input distribution S is further split into metadata transmission S. M With data block transfer S D ; Construct the order constraint C1 within the micro-batch according to the order of gate calculation G, input distribution S, expert calculation E, result aggregation R, and combined output C; The gated computation G includes the first gated computation task G i and the second gated computation task G i+1 , according to the first gated computing task G i and the second gated computation task G i+1 In the order of the micro-batches between the i-th micro-batch and the i+1-th micro-batch, we construct the cross-micro-batch isomorphic task order constraint C2. For n micro-batch gated computing tasks (G1, G2…, G n ), with n input radiating (S1, S2…, S n ), according to the n input distribution (S1, S2…, S n ) and then perform result aggregation R to build the global constraint C3 of the communication task; According to the metadata transmission within the same micro-batch i, Prioritize block transfers When crossing micro-batches i and i+1, the i-th micro-batch data block is transferred Prioritizes metadata transmission over the i+1th micro-batch Before completion, build metadata-data dependency constraint C4; For n micro-batch gated computations G and expert computations E, we use the cross-micro-batch homogeneous task order constraint C2, take advantage of the sequential execution characteristics within the stream, and assign them to independent GPU streams for execution, thus building an intra-stream constraint implementation. The computation task executed later in the same micro-batch uses events to depend on the task executed earlier. In this case, the order constraint C1 within the micro-batch is followed. The result aggregation task of all computation tasks uses events to depend on the input distribution task. In this case, the global constraint C3 of the communication task is followed. The metadata of the i+1th micro-batch is transmitted. Using events depends on the i-th micro-batch data block transmission At this time, we follow the metadata-data dependency constraint C4 and build inter-stream constraint implementation to establish event synchronization in different stream tasks and enforce the execution order of different stream tasks. During the task submission phase, the reasoning process of the hybrid expert model dynamically decides the task submission timing and performs checks based on the intra-micro-batch order constraint C1, the cross-micro-batch isomorphic task order constraint C2, the global constraint C3 of the communication task, the metadata-data dependency constraint C4, the intra-stream constraint implementation, and the inter-stream constraint implementation.
2. The method for arranging pipeline tasks in hybrid expert model reasoning according to claim 1, wherein: The i-th micro-batch gating calculation G in the n micro-batch gating calculation G i Submit to Stream G Flow, gated computation G for the i-th micro-batch i After completion, the i+1th micro-batch gate calculation G is automatically triggered i+1 , construct the in-stream constraint implementation.
3. The method for arranging pipeline tasks in hybrid expert model reasoning according to claim 1, wherein: When the result aggregation task of all the computing tasks follows the global constraint C3 of the communication task, the ith micro-batch input is distributed S i and the i-th mini-batch expert calculates E i , S is distributed in the i-th micro-batch input i Record the event cudaEventRecord(event Si ,stream S ), in the i-th micro-batch expert calculates E i Insert event before submitting wait cudaStreamWaitEvent(stream E ,event Si ), to ensure that the i-th micro-batch expert calculates E i S is distributed only on the i-th micro-batch input i After submission, it is started to implement the aforementioned mandatory constraints on the execution order of different flow tasks.
4. The method for arranging pipeline tasks in hybrid expert model reasoning according to claim 3, wherein: The check includes: when the i-th mini-batch expert calculates E i When the software layer is ready, verify that the i-th micro-batch input emits S i Whether the event completion mark in the flow or the implicit synchronization signal of the communication interface has been met, if so, the task is submitted; if not, the submission is delayed until the dependency is released.