Inference method, system, device and medium of dynamic routing hybrid expert model

Abstract

The application discloses a dynamic route hybrid expert model reasoning method, system, device and medium, which are corresponding solutions, and in the solutions: through automatic parallel strategy search, the application can divide the model into pipeline stages with balanced running time, reduce the idle rate of the calculation unit, and effectively improve the execution efficiency of the calculation unit; and the automatic search process of the application can reduce the maximum pipeline stage running time as much as possible under the condition of meeting the memory limit, so that the reasoning performance of the model is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model reasoning system design technology, and in particular to a reasoning method, system, device and medium for a dynamic routing hybrid expert model. Background Technology

[0002] The Mixture of Experts (MoE) model architecture has become the mainstream paradigm for extending large language models (LLMs) based on transformer architectures. It achieves a significant increase in model parameter size with only a small increase in computation by replacing the feed-forward network (FFN) module in dense models with a hybrid expert module consisting of several independent expert networks and a learnable routing mechanism.

[0003] Existing hybrid expert layers often employ a static Top-k routing mechanism: selecting the best k (a fixed number) expert networks for each token to participate in the computation. To further optimize computational efficiency and achieve more refined resource allocation, researchers have proposed various dynamic routing strategies. These mechanisms analyze the characteristics of each token and allocate different numbers of experts to different tokens, often reducing the computational overhead of the model without affecting its performance. Another characteristic of dynamic routing strategies is that the computational overhead of the hybrid expert modules within different transformer layers often varies and exhibits a certain fixed trend with the number of layers.

[0004] In a distributed system consisting of multiple computing nodes, inference of large language models typically employs a hybrid parallelism strategy. A common approach is to first divide the model into multiple stages using pipeline parallelism (PP) and deploy them across different unit groups, then apply tensor parallelism (TP) or expert parallelism (EP) within each stage to improve system throughput. However, for hybrid expert models employing dynamic routing strategies, the naive hybrid parallelism strategy cannot effectively balance the computational and memory loads between stages due to the uneven computational load among the hybrid expert modules, resulting in inefficient model inference performance.

[0005] Therefore, in order to address this challenge, it is necessary to propose an inference method suitable for dynamic routing hybrid expert models to improve the efficiency of computing nodes and the inference performance of the model.

[0006] In view of this, the present invention is hereby proposed. Summary of the Invention

[0007] The purpose of this invention is to provide a reasoning method, system, device, and medium for a dynamic routing hybrid expert model, which can handle the non-uniform computational load introduced by dynamic routing strategies, thereby effectively improving the inference efficiency of the model.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A reasoning method for a dynamic routing hybrid expert model includes:

[0010] The modules contained in each transformer layer of the dynamic routing hybrid expert model are used as basic units to form a sequence of modules to be partitioned; wherein, the modules include attention modules and hybrid expert modules;

[0011] In the sequence of modules to be partitioned, based on the running time and memory usage of each module under different pipeline partitioning methods and different parallel strategies, search for the pipeline partitioning method and the corresponding parallel strategy that meet the preset memory limit and take the minimum maximum running time among all pipeline stages.

[0012] The dynamic routing hybrid expert model is controlled to perform parallel inference using the searched pipeline partitioning method and the corresponding parallel strategy.

[0013] A dynamic routing hybrid expert model inference system for implementing the aforementioned method includes:

[0014] A module sequence construction unit is used to form a module sequence to be divided by taking the modules contained in each transformer layer of the dynamic routing hybrid expert model as basic units; wherein, the module includes attention modules and hybrid expert modules;

[0015] The pipeline partitioning scheme and parallel strategy search unit is used to search for the pipeline partitioning scheme and parallel strategy in the sequence of modules to be partitioned, based on the running time of each module under different pipeline partitioning schemes and different parallel strategies, and combined with the memory usage of each module, to find the pipeline partitioning scheme and the corresponding parallel strategy that minimizes the maximum running time of all pipeline stages under the preset memory limit.

[0016] The parallel inference unit is used to control the dynamic routing hybrid expert model to perform parallel inference using the searched pipeline partitioning method and the corresponding parallel strategy.

[0017] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0020] As can be seen from the technical solution provided by the present invention, by using automated parallel strategy search, the present invention can divide the model into pipeline stages with balanced runtime, reduce the idle rate of computing nodes, and thus effectively improve the execution efficiency of computing nodes. Furthermore, the automated search process of the present invention, through fine-grained parallel strategy partitioning, can minimize the runtime of the largest pipeline stage while meeting memory constraints, thereby improving the inference performance of the model. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a reasoning method for a dynamic routing hybrid expert model provided in an embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the hybrid expert model structure provided in an embodiment of the present invention.

[0024] Figure 3 This diagram illustrates the average number of experts at each layer for different datasets using the Top-K dynamic routing strategy provided in this embodiment of the invention.

[0025] Figure 4 This is a schematic diagram of data parallelism provided for an embodiment of the present invention.

[0026] Figure 5 This is a schematic diagram of a parallel pipeline provided for an embodiment of the present invention.

[0027] Figure 6 This is a schematic diagram of expert parallelism based on all-to-all communication provided for an embodiment of the present invention.

[0028] Figure 7 This is a schematic diagram of tensor parallelism provided for an embodiment of the present invention.

[0029] Figure 8 This is a schematic diagram of the communication process without using the DPMoE strategy, provided as an embodiment of the present invention.

[0030] Figure 9 This is a schematic diagram of the communication process when using the DPMoE strategy, as provided in an embodiment of the present invention.

[0031] Figure 10 This is a schematic diagram of an inference system for a dynamic routing hybrid expert model provided in an embodiment of the present invention.

[0032] Figure 11 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0034] First, the following explanations are provided for the terms that may be used in this article:

[0035] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0036] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0037] The following provides a detailed description of the reasoning method, system, device, and medium for a dynamic routing hybrid expert model provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0038] Example 1

[0039] This invention provides an inference method for a dynamic routing hybrid expert model. The dynamic routing hybrid expert model includes multiple transformer layers, each transformer layer comprising an attention module and a hybrid expert module employing a dynamic routing strategy, arranged sequentially. The dynamic routing strategy activates different numbers of expert networks for different input terms (derived from text), such as... Figure 1 As shown, the method mainly includes the following steps:

[0040] Step 1: Take the modules contained in each transformer layer of the dynamic routing hybrid expert model as basic units to form a sequence of modules to be divided.

[0041] In this embodiment of the invention, the module includes an attention module and a hybrid expert module. For example, when the total number of transformer layers is N, the length of the module sequence to be divided is 2N.

[0042] Step 2: Search for pipeline segmentation schemes and parallel strategies.

[0043] In this embodiment of the invention, in the sequence of modules to be divided, based on the running time and memory usage of each module under different pipeline partitioning methods and different parallel strategies, the pipeline partitioning method and the corresponding parallel strategy that meet the preset memory limit and whose maximum running time among all pipeline stages is the minimum value are searched.

[0044] In this embodiment of the invention, the methods for obtaining the running time and memory usage of each module include: adding timing markers during the operation of each converter layer of the dynamic routing hybrid expert model; using analysis tools to parse the timing markers generated during the operation to obtain the running time of each module under different pipeline partitioning methods and different parallel strategies; calculating the memory usage of each module according to the given configuration file of the dynamic routing hybrid expert model, and then calculating the memory usage of each module on each computing node in combination with the corresponding parallel strategy; wherein, each pipeline stage is executed by different computing nodes.

[0045] For example, assuming the module memory usage is 2GB (gigabytes) as obtained from the configuration file, this module will be allocated to 2 compute nodes. If the two compute nodes use data parallelism, the memory usage of each compute node will be 2GB. If tensor parallelism or expert parallelism is used, the memory usage will be 1GB.

[0046] In this embodiment of the invention, when the pipeline partitioning method and parallel strategy for finding the minimum maximum runtime among all pipeline stages that satisfies the preset memory limit include: defining the runtime of pipeline stage j as... Where Dur is the runtime function. This indicates the modules included in pipeline stage j. Let r be the parallelism strategy for each module included in pipeline stage j, corresponding to tensor parallelism, expert parallelism, and data parallelism, respectively; define the number of computing nodes used in pipeline stage j as r, and the memory usage as r. Where Men is a memory usage function; combined with the runtime... With memory usage An optimization problem is constructed, and by solving the optimization problem, the pipeline partitioning method and the corresponding parallel strategy are obtained when the maximum running time of all pipeline stages is minimized under the preset memory limit.

[0047] In this embodiment of the invention, the optimization problem is expressed as:

[0048] ;

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] Among them, the set of integers set of integers set of integers ,symbol For the definition of symbols, S represents pipeline parallelism, 2N represents the total number of modules, and N represents the total number of converter layers; All are optimization variables. The parallel strategies for each module included in all pipeline stages correspond to tensor parallelism, expert parallelism, and data parallelism, respectively. , , , The symbol 'B' represents the split point of each pipeline stage, containing the sequence number of the last module in each pipeline stage. , Used to restrict pipeline stage 1 to start from module 1. This is the sequence number of the last module in pipeline stage 1, and the corresponding module in pipeline stage 2 is the... Module 1 to Module 2 The module corresponding to pipeline stage S is the module number 1. Module 1 to Module 2 Each module , , This corresponds to the last module in pipeline stage 2, pipeline stage S-1, and pipeline stage S; for pipeline stage j, its included modules are represented as follows. That is, the right side of the equation represents the first... Module 1 to Module 2 One module; In the equation, the right side represents the first stage (j) of the pipeline. Module 1 to Module 2 The tensor parallelism of each module In the equation, the right side refers to the first stage j included in the pipeline. Module 1 to Module 2 The degree of expert parallelism for each module In the equation, the right side refers to the first stage j included in the pipeline. Module 1 to Module 2 The data parallelism of each module; , The corresponding symbols represent the minimization function and the maximization function; st is the constraint symbol; M is the preset memory limit; Corresponding representation The kth element, i.e., the kth module Tensor parallelism, expert parallelism, and data parallelism. Represents the k-th module The If the pipeline is divided, r computing nodes are needed. For pipeline stage j, all its modules share r computing nodes, so the number of computing nodes used by pipeline stage j is r; mod is the modulo operation, representing taking the remainder. This means that when the remainder of k divided by 2 is 1, that is, when k is odd, the k-th module... Expert parallelism When k is 1, the corresponding modules when k is odd are all attention modules.

[0059] In this embodiment of the invention, the pipeline partitioning method and the corresponding parallel strategy obtained by solving the optimization problem to satisfy the preset memory limit and the minimum maximum running time among all pipeline stages include: using a binary search algorithm, adjusting the upper and lower limits of the pipeline stage time to find the minimum maximum running time among the pipeline stages, thereby obtaining the corresponding pipeline partitioning method and the corresponding parallel strategy.

[0060] The algorithm flow for binary search is as follows:

[0061] Step A1: Initial time, set the lower limit Set the upper limit for the preset minimum value. This is the preset maximum value; for example, the upper limit can be set. Let Dur(L) be the time required to run the entire model.

[0062] Step A2: At the current moment, determine the upper limit. and lower limit If the difference is greater than the preset solution accuracy, proceed to step A3; otherwise, proceed to step A4.

[0063] Step A3: Calculate the upper limit and lower limit The mean is used as the median. ; at the median Under the constraints, solve for a pipeline partitioning scheme and its corresponding parallel strategy (the maximum runtime of each pipeline stage is less than...). If the solution is successful, the corresponding pipeline partitioning method and the corresponding parallel strategy will be taken as the current optimal result, and the median will be set as the optimal value. As an upper limit If the solution fails, the median will be... As a lower limit Then, proceed to step A2.

[0064] Step A4: Set the upper limit Output the current optimal result as the minimum of the maximum runtime in the pipeline stage.

[0065] In this embodiment of the invention, in step A3 above, the median Under certain constraints, solving for a pipeline partitioning scheme and its corresponding parallelization strategy includes:

[0066] Step B1, Initialize parameters: Set the left boundary (left) and right boundary (right) of the current pipeline stage to the first module, set the number of current pipeline stages (stages) to 0, set the parallel strategy of the pipeline stage and all pipeline stages to an empty set, and set the parallel strategy of the current pipeline stage to empty; where left boundary (left) and right boundary (right) refer to the first module and the last module of the current pipeline stage, respectively.

[0067] Step B2: Determine if left < 2N. If yes, proceed to step B3; otherwise, proceed to step B4. Here, 2N is the total number of modules.

[0068] Step B3: Using the left boundary (left), right boundary (right), the number of computation nodes in the current pipeline stage, and the preset memory limit M, solve the knapsack problem to obtain the optimal running time and corresponding parallel strategy for the current pipeline stage (the left-th to right-th modules); then, determine whether the constraint condition is satisfied: right If the current right boundary is within the model (i.e., whether it is 2N), and the current pipeline stage's runtime is less than or equal to the preset time limit Q, then increment the right boundary (right) by 1 and save the corresponding parallel strategy to the current pipeline stage's parallel strategy. If not, check if the current pipeline stage's parallel strategy is empty. If it is empty, output false and end the algorithm. If it is not empty, add (left, right - 1) to the pipeline stage, i.e., go back to the previous pipeline stage that satisfies the constraints, with its first module being the left boundary (left) and its last module being the right boundary (right - 1). Add the parallel strategy corresponding to the previous pipeline stage that satisfies the constraints to the parallel strategies of all pipeline stages, assign the right boundary (right) to the left boundary (left), increment the number of pipeline stages currently divided by 1, and reset the current pipeline stage's parallel strategy to empty. Then, return to step B2.

[0069] Step B4: Determine if the current number of pipeline stages (stages) is less than or equal to the pipeline parallelism S. If yes, output "true"; otherwise, output "false". A true output means that the pipeline partitioning method and the corresponding parallelism strategy satisfy the median. The output being false indicates that the pipeline partitioning method and the corresponding parallel strategy do not satisfy the median. Restrictions.

[0070] Step 3: Use the searched pipeline partitioning method and corresponding parallel strategy to control the dynamic routing hybrid expert model to perform parallel inference.

[0071] In this embodiment of the invention, the dynamic routing hybrid expert model is deployed in an inference framework. The inference framework is configured using the searched pipeline partitioning method and the corresponding parallel strategy, and then the inference framework controls the dynamic routing hybrid expert model to perform parallel inference.

[0072] In the above-described solution provided by the embodiments of the present invention, by using automated parallel strategy search, the present invention can divide the model into pipeline stages with balanced runtime, reduce the idle rate of computing nodes, and thus effectively improve the execution efficiency of computing nodes; furthermore, the automated search process of the present invention, through fine-grained parallel strategy partitioning, can minimize the runtime of the largest pipeline stage while meeting memory constraints, thereby improving the inference performance of the model.

[0073] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0074] I. Introduction to the principles of the overall solution.

[0075] The method provided by this invention is based on the analysis of the computational and memory load (running time and memory usage) of each module of the dynamic routing hybrid expert model. The modules are divided and deployed in a distributed system for parallel execution to achieve high-performance inference. The distributed system is hardware configured with an inference framework for executing the calculations of each module in the dynamic routing hybrid expert model. The logic involved here falls within the scope of conventional technology and will not be elaborated upon.

[0076] In this embodiment of the invention, the hybrid expert model is a large language model containing multiple transformer layers. Each transformer layer includes an attention module and a hybrid expert module. Its specific structure can be found in [reference needed]. Figure 2 The aforementioned dynamic routing hybrid expert model refers to a hybrid expert module employing a dynamic routing strategy. Internally, the hybrid expert module uses a dynamic routing strategy to assign a variable number of experts to different tokens for computation. In two identical input token sequences, for each position, the dynamic routing strategy assigns the same expert to the token at that position, or the expert is sampled based on a non-completely random probability distribution.

[0077] Text, as input to large language models, exhibits a specific statistical distribution. Applying this distribution to dynamically parallel hybrid expert models results in little difference in the average number of experts assigned to different input lexical sequences by the dynamic routing strategy of fixed hybrid expert modules. However, the average number of experts assigned by the dynamic routing strategies of different hybrid expert modules within the same model will differ. For example... Figure 3 As shown, in the model using the Top-p dynamic routing strategy, the average number of experts for dynamic routing in each hybrid expert module is similar for different datasets, and it will show a specific trend as the hierarchical order changes. Figure 3 In Chinese: BoolQ is a dataset released by the University of Washington and Google for answering yes / no questions, containing 15,942 examples; CNN-DailyMail is a dataset containing over 300,000 English news articles written by journalists from CNN and the Daily Mail, released by Stanford University and Google; HellaSwag is a dataset released by the University of Washington and the Allen Institute for Artificial Intelligence for testing commonsense natural language reasoning. In typical hybrid expert modules, a score for each word is calculated for each expert network (the sum of the scores of all expert networks is 1). In the Top-p dynamic routing strategy, the highest-scoring expert networks are selected sequentially until the sum of the scores of the selected expert networks first exceeds a certain threshold p. That is, in this case, the number of expert networks selected is uncertain.

[0078] The distributed system described in this embodiment of the invention consists of multiple computing nodes, which are interconnected and communicate with each other via a network. Each computing node has independent memory and includes, but is not limited to, a single central processing unit (CPU), a single graphics processing unit (GPU), a single tensor processing unit (TPU), and a single neural processing unit (NPU) found in a server.

[0079] Existing technologies often combine multiple parallel methods to complete inference on hybrid expert models in distributed systems. Commonly used parallel methods for hybrid expert models include data parallelism (DP), pipeline parallelism (PP), expert parallelism (EP), and tensor parallelism (TP).

[0080] like Figure 4 As shown, data parallelism deploys the complete model in different computing node working groups with the same internal parallel strategy, and uses a load balancer to divide the input received by the entire system and distribute it to each data parallel working group.

[0081] like Figure 5 As shown, pipelined parallelism divides continuous layers within the model into different stages and assigns them to different workgroups to perform inference on the input in a pipeline manner. In pipelined parallelism, the workgroups corresponding to adjacent stages transmit intermediate computation results through point-to-point (P2P) communication.

[0082] like Figure 6 As shown, expert parallelism completes the computation of the hybrid expert module by assigning each expert network within the hybrid expert module to different workgroups and using All Reduce or All to All communication to distribute the input and collect the expert computation results based on the input routing results.

[0083] like Figure 7 As shown, tensor parallelism divides and distributes the weights within each layer of the model to each computation node, and collects the computation results of each node at the end of each layer through full reduction communication, so as to achieve mathematically equivalent computation as without tensor parallelism.

[0084] By executing one parallelism mode within a workgroup that uses a specific parallelism mode, different parallelism modes can be combined to achieve more granular resource allocation.

[0085] Pipeline parallelism offers significant advantages over other parallel methods during inference execution due to its lower communication overhead. However, the computational imbalance introduced by dynamic routing can lead to inefficiencies when using a hybrid expert model with dynamic routing and pipelined parallel inference. If each stage is divided into the same number of layers, the computation time of each stage will differ, resulting in "bubbles" in the pipeline (i.e., certain computational nodes are idle due to waiting), thus reducing efficiency. If the model is simply divided to average the computation time of each stage, the model weights and key-value cache memory usage will differ across stages, leading to memory waste.

[0086] To ensure compatibility between dynamic routing and pipelined parallelism, this invention employs fine-grained stage partitioning and flexible module-level parallel allocation. For pipelined parallelism, existing inference methods often partition the model using transformer layers as the basic unit, while this invention partitions the model using modules (attention modules and hybrid expert modules). This partitioning method better balances the computation time of each pipeline stage. Furthermore, within each pipeline stage, existing inference methods often employ a uniform parallel strategy for both types of modules, while this method uses different parallel strategies for different hybrid expert modules depending on the situation.

[0087] Specifically, inspired by the DP Attention strategy used in the SGLang framework (a high-performance inference and service framework for large language models and visual language models), which is a key performance optimization strategy whose core idea is to avoid the problem of duplicate storage of key-value caches in specific models by using data parallelism on the attention module, and also to reduce the communication requirements and overhead of the attention module, this method proposes the DPMoE strategy (using data parallelism on hybrid expert modules to reduce the communication requirements and overhead of hybrid expert modules). By increasing the data parallelism of specific hybrid expert modules (i.e., hybrid expert modules with a data parallelism greater than 1 in the solution of the knapsack problem), that is, copying the weights of the module to the computing node in the data parallel dimension, the size of other parallel (tensor parallelism and expert parallelism) groups is reduced by using idle memory, thereby reducing the corresponding communication overhead. Figure 8 and Figure 9 This demonstrates the difference between using DPMoE and not using DPMoE for a single converter layer in a distributed system consisting of four compute nodes. Figure 8 The communication process when the DPMoE strategy is not used. Figure 9 This describes the communication process when using the DPMoE strategy. After using DPMoE, the communication operations spanning four compute nodes in the hybrid expert module (left side) are transformed into communication operations spanning two compute nodes (right side), thereby reducing communication overhead.

[0088] By utilizing the partitioning strategy and parallel methods described above, this invention can minimize the runtime of the largest pipeline stage without using additional resources, thereby improving the end-to-end performance of model inference.

[0089] II. Detailed introduction of the plan.

[0090] 1. Overall execution process.

[0091] In this embodiment of the invention, performance analysis of each module is required to collect its runtime and memory requirements under different parallel strategies. Based on the collected results, the invention uses a search algorithm to divide the dynamic routing hybrid expert model into different pipeline stages at the module level, and assigns a parallel strategy to each module. Then, a reasoning framework is used with the results of the search algorithm as the configuration to perform parallel inference.

[0092] 2. Specific details.

[0093] (1) Performance analysis of the module.

[0094] To perform performance analysis on each module, this invention utilizes the Application Programming Interface (API) provided by the computing node device to add timing markers to the execution flow of each converter layer. Then, using an inference framework, the model's inference is run under various possible parallel strategies. Representative inputs are distributed to a dynamic routing hybrid expert model to obtain runtime data that approximates real-world conditions. In this embodiment, analysis tools are used to parse the timing markers generated during execution to obtain the runtime of each module under different parallel strategies. The memory usage of each module under each parallel strategy can be calculated based on the model configuration file.

[0095] (2) Segmentation scheme and parallel strategy search.

[0096] Given a dynamic routing hybrid expert model with N transformer layers, define a length of The sequence of modules to be divided (referred to as the sequence) For all its main modules, of which and Let i represent the attention module and the hybrid expert module of the i-th transformer layer, i=1,…,N. For ease of introduction later, all modules will be uniformly numbered as follows: For the k-th module, let it be denoted as Let k = 1, 2, ..., 2N. When k is odd, it represents the attention module; when k is even, it represents the hybrid expert module. Let the pipeline parallelism be S, meaning the sequence L will be divided into S stages: ,in, Let j be the modules contained in pipeline stage j, where j = 1, ..., S.

[0097] Define a series of optimization variables These serve as the dividing points for each stage, and Therefore, the modules contained in pipeline stage j can be represented as follows: This represents the module included in pipeline stage j as the first... Module 1 to Module 2 Each module defines a series of optimization variables. , , This defines the tensor parallelism, expert parallelism, and data parallelism for each module. Similar to the definitions above, , , ,in, , , The parallel strategies for each module included in pipeline stage j are, respectively, tensor parallelism, expert parallelism, and data parallelism. The meanings of the symbols involved in this part can be found in the previous explanation, and will not be repeated here.

[0098] For pipeline stage j, its runtime is affected by its internal modules and the module parallelism strategy; therefore, the runtime of this pipeline stage is expressed as... If a module Parallelism If it is divided, then it needs to be divided. There are [number] computation nodes. To simplify the problem, the number of computation nodes used in each pipeline stage is limited to the same number. Therefore, for all converter layer indices i, the number of computation nodes involved in its internal attention module and hybrid expert module is the same, i.e., [number] computation nodes. Where P is the total number of computing nodes in the distributed system. Also, for pipeline stage j, all its modules share r computing nodes; therefore, pipeline stage j has r computing nodes, and the memory usage on those r computing nodes is expressed as... The memory usage on each computing node will not exceed the preset upper bound M (preset memory limit). It is the symbol for the set of real numbers.

[0099] Based on the above definition, for a specific pipeline parallelism S, this invention formalizes the search for the partitioning scheme and parallel strategy into the following optimization problem: minimizing the maximum pipeline stage time while satisfying memory constraints.

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] ;

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] Among them, the set of integers set of integers set of integers ,symbol This is a definition symbol. Furthermore, the meanings of the relevant symbols have already been explained in detail above, and will not be repeated here.

[0111] Since the available pipeline parallelism in a given distributed system is very limited, this invention enumerates possible and commonly used pipeline parallelisms, solves the above optimization problem, and determines the final pipeline parallelism and corresponding module-level parallelism strategy by comparing different optimization results. Through performance analysis, this invention can accurately estimate the runtime. and memory usage This ensures the accuracy of the solution results.

[0112] However, to quickly solve the above optimization problem and avoid the non-polynomial time complexity caused by a huge search space, this invention adopts the binary search-based algorithm shown in Table 1 (hereinafter referred to as the main algorithm). The main algorithm adjusts the pipeline time limit. and lower limit The minimum pipeline stage time that can be satisfied is obtained by binary search, thus obtaining the optimal pipeline partitioning method and the corresponding parallel strategy.

[0113] Table 1: Algorithms based on binary search (main algorithm)

[0114]

[0115] The main algorithm determines in line 4 whether the pipeline stage time is satisfactory under the current configuration using the algorithm shown in Table 2 (hereinafter referred to as the sub-algorithm). Since the sub-algorithm aims to minimize the pipeline stage time while meeting memory constraints, which is similar to solving the two-dimensional knapsack problem, this invention uses a "knapsack problem solving" sub-process in line 3 of the sub-algorithm to search for the optimal module-level parallelism strategy.

[0116] Table 2: Sub-algorithms

[0117]

[0118] By using dynamic programming, the sub-algorithm can achieve a minimum time complexity of 100%. Correspondingly, the time complexity of the main algorithm is... Where Q is the time limit, which is the maximum time to run the entire dynamic routing hybrid expert model. The algorithm has asymptotic time complexity, which corresponds to the asymptotic upper bound of the running time.

[0119] By invoking the main algorithm, this invention can search for an optimal module-level parallel strategy and hand it over to the inference framework for execution. Considering that the algorithms shown in Tables 1 and 2 have already been described in detail in the tables, and that a corresponding step description has been provided above, they will not be repeated here.

[0120] (3) Apply search results.

[0121] The module-level parallel strategy obtained by this invention needs to be implemented by a specific inference framework. To achieve this, in addition to common model inference requirements, the inference framework also needs to support dynamic routing of hybrid expert models and module-level parallel strategies. Commonly used inference frameworks need to implement dynamic routing strategies and support setting up separate parallel communication groups for each module. Within the execution flow of each module, independent parallel communication groups can be used to complete expert parallelism and tensor parallelism; to switch data parallelism between different modules, the inference framework also needs to complete the collection and redistribution of input data between modules; to pipeline the model using modules as the basic unit, the inference engine may need to allow for the absence of specific modules within the transformer layer.

[0122] III. Specific Examples.

[0123] In this example, on a server with eight NVIDIA A40 graphics processors (compute nodes), based on the open-source inference framework SGLang (version v0.4.9.post1), the method described in this invention is used to infer the modified dynamic routing strategy of the Qwen2-57B-A14B hybrid expert model. Qwen2 represents the second-generation Qwen2 hybrid expert model, 57B indicates 57 billion parameters, and A14B indicates 14 billion activation parameters. It should be noted that this invention is not limited to specific hardware facilities, inference frameworks, or dynamic routing hybrid expert model structures.

[0124] First, the following modifications need to be made to SGLang:

[0125] (1) Use the NVTX (NVIDIA Tool Extension Library) programming interface provided by NVIDIA to mark the execution process of the model's attention module and hybrid expert module so that the analysis tool can extract data.

[0126] (2) When the attention module and the hybrid expert module are initialized, the module-level parallel strategy provided by the present invention is read to determine the parallel communication groups of the module.

[0127] (3) During the initialization of each converter layer, the pipeline partitioning scheme provided by this invention is read to determine whether it contains two complete modules. Furthermore, a method for executing any module separately is provided for the converter layer, that is, the relevant code is modified: after the computing node reads the pipeline stage partitioning scheme, if a converter layer contains only an attention module (which can only appear in the last converter layer of the pipeline stage) in the stage that the computing node is responsible for, then only the attention module weight of the converter layer is loaded, and only the calculation of the attention module is completed when the converter layer is executed, and then the result is handed over to the computing node responsible for the next pipeline stage; if a converter layer contains only a hybrid expert module (which can only appear in the first converter layer of the pipeline stage) in the stage that the node is responsible for, then only the hybrid expert module weight of the converter layer is loaded, and when the converter layer is executed, after receiving the calculation result from the previous pipeline stage, the result is directly input to the hybrid expert module to complete the calculation of the converter layer, and the attention module of the converter layer is skipped.

[0128] (4) When the transformer layer is executed, it divides the output of the attention module as needed according to the change of data parallelism and rearranges it into the input of the hybrid expert module.

[0129] (5) Provide dynamic routing strategies and modify the kernel functions (such as moe_align_block_size and pre_reorder_triton_kernel) in the hybrid expert module to rearrange the routing results to adapt to dynamic routing strategies. Considering that the way to modify the kernel functions in this part is a conventional technique, it will not be elaborated.

[0130] Then, after preparing representative input data, the modified SGLang inference framework needs to be run under various parallel configurations to perform model inference on these input data. During this process, NVIDIA NsightSystems (NVIDIA's system-level performance analysis tool) is used to record the time taken during inference and then the data is analyzed by the analysis tool.

[0131] Parsing the runtime and memory usage Then, the partitioning scheme and parallel strategy search algorithm described in this invention is run to obtain the optimal partitioning scheme and parallel strategy (i.e., the pipeline partitioning method and corresponding parallel strategy searched above). Finally, the partitioning scheme and parallel strategy are provided to the modified SGLang inference framework to achieve high-performance model inference.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0133] Example 2

[0134] This invention also provides an inference system for a dynamic routing hybrid expert model, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 10 As shown, the system mainly includes:

[0135] A module sequence construction unit is used to form a module sequence to be divided by taking the modules contained in each transformer layer of the dynamic routing hybrid expert model as basic units; wherein, the module includes attention modules and hybrid expert modules;

[0136] The pipeline partitioning scheme and parallel strategy search unit is used to search for the pipeline partitioning scheme and parallel strategy in the sequence of modules to be partitioned, based on the running time of each module under different pipeline partitioning schemes and different parallel strategies, and combined with the memory usage of each module, to find the pipeline partitioning scheme and the corresponding parallel strategy that minimizes the maximum running time of all pipeline stages under the preset memory limit.

[0137] The parallel inference unit is used to control the dynamic routing hybrid expert model to perform parallel inference using the searched pipeline partitioning method and the corresponding parallel strategy.

[0138] Since the specific technical details involved in the system have been described in detail in the previous embodiments, they will not be repeated here.

[0139] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0140] Example 3

[0141] The present invention also provides a processing device, such as Figure 11 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0142] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0143] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0144] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0145] The output device can be a display terminal;

[0146] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0147] Example 4

[0148] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0149] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0150] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. An inference method of a dynamic routing hybrid expert model, the dynamic routing hybrid expert model comprising a plurality of transformer layers, each transformer layer comprising an attention module and a hybrid expert module with a dynamic routing strategy arranged in sequence, the dynamic routing strategy activating different numbers of expert networks for different input tokens, characterized in that, The application comprises the following steps: The modules contained in each transformer layer of the dynamic routing hybrid expert model are taken as basic units to form a module sequence to be divided. In the module sequence to be divided, according to the running time and memory usage of each module when different parallel strategies are used under different pipeline division manners, a pipeline division manner and a corresponding parallel strategy are searched out, which satisfy a preset memory limit and make the maximum running time in all pipeline stages take the minimum value. The dynamic routing hybrid expert model is controlled to perform parallel reasoning by using the searched pipeline division manner and corresponding parallel strategy.

2. The reasoning method of a dynamic routing hybrid expert model according to claim 1, wherein, The running time and memory usage of each module are obtained in the following manner: Timing marks are added during the running process of each transformer layer of the dynamic routing hybrid expert model. An analysis tool is used to analyze the timing marks generated during the running process to obtain the running time of each module when different parallel strategies are used under different pipeline division manners.

3. The reasoning method of a dynamic routing hybrid expert model according to claim 1, wherein, According to the configuration file of the given dynamic routing hybrid expert model, the memory usage of each module is calculated, and the memory usage of each module on each computing node is calculated in combination with the corresponding parallel strategy; wherein the computing node is responsible for executing the corresponding pipeline stage. define the running time of pipeline stage j as wherein Dur is a running time function, denotes the modules contained in pipeline stage j, is the parallel strategy of each module contained in pipeline stage j, corresponding to tensor parallelism degree, expert parallelism degree and data parallelism degree in turn. Let r be the number of compute nodes used by pipeline stage j, and let M be the memory footprint where Men is the memory footprint function. The running time is combined The memory usage , and an optimization problem is constructed. By solving the optimization problem, a corresponding pipeline division manner and a corresponding parallel strategy are obtained, which satisfy that the maximum running time in all pipeline stages is minimized under a preset memory limit.

4. The reasoning method of a dynamic routing hybrid expert model according to claim 3, wherein, The search of the pipeline division manner and the corresponding parallel strategy, which satisfy the preset memory limit and make the maximum running time in all pipeline stages take the minimum value, comprises the following steps: ； ； ； ； ； ； ； ； ； ； Wherein, the integer set , the integer set , the integer set , the symbol is a definition symbol, S is a pipeline parallelism, 2N is a total number of modules, N is a total number of transformer layers, and j is an index symbol of a pipeline stage; All are optimization variables, is a parallel strategy of each module contained in all pipeline stages, which corresponds to a tensor parallelism, an expert parallelism, and a data parallelism in turn; , , , is a connection symbol; B represents a split point of each pipeline stage, and contains a serial number of the last module of each pipeline stage, , is used to limit the pipeline stage 1 to start from the first module, is a serial number of the last module of the pipeline stage 1, the pipeline stage 2 corresponds to the module from the first module to the first module, the pipeline stage S corresponds to the module from the first module to the first module, , , corresponding to the last module in the pipeline stage 2, the pipeline stage S-1, and the pipeline stage S; for the pipeline stage j, the module contained is represented as , that is, the right side of the equation represents the first module to the first module; , the right side of the equation represents the first module to the first module contained in the pipeline stage j, each corresponding to a tensor parallelism, , the right side of the equation refers to the first module to the first module contained in the pipeline stage j, each corresponding to an expert parallelism, , the right side of the equation refers to the first module to the first module contained in the pipeline stage j, each corresponding to a data parallelism; , corresponding to represent a minimization function and a maximization function; s.t. is a constraint symbol; M is a preset memory limit; corresponding to represent the kth element of , that is, the tensor parallelism, the expert parallelism, and the data parallelism of the kth module , represent the kth module The If the pipeline is divided, r computing nodes are needed. For pipeline stage j, all its modules share r computing nodes, so the number of computing nodes used by pipeline stage j is r; mod is the modulo operation, representing taking the remainder. This means that when the remainder of k divided by 2 is 1, that is, when k is odd, the k-th module... Expert parallelism When k is 1, the corresponding modules when k is odd are all attention modules.

5. The reasoning method of a dynamic routing hybrid expert model according to claim 3 or 4, characterized in that, The optimization problem is expressed as: The pipeline division manner and the corresponding parallel strategy, which satisfy the preset memory limit and make the maximum running time in all pipeline stages take the minimum value, are obtained by solving the optimization problem, and comprise the following steps: Step Al, Initial time, set lower limit upper limit set as preset minimum value upper limit set as preset maximum value Step A2: At the current moment, determine the upper limit. and lower limit If the difference is greater than the preset solution accuracy, proceed to step A3; otherwise, proceed to step A4. Step A3, calculating upper limit and lower limit ; under the limit of median , a pipeline division mode and corresponding parallel strategy are solved, if the solving is successful, the corresponding pipeline division mode and corresponding parallel strategy are taken as the current optimal result, and median is taken as upper limit , if the solving fails, median is taken as lower limit , then, step A2 is entered.​ Step A4, upper limit The current optimal result is output as the minimum of the maximum runtimes in the pipeline stages.

6. The reasoning method of a dynamic routing hybrid expert model according to claim 5, wherein, The in median Under the restriction of the in median Solving a pipeline partitioning and corresponding parallel strategy includes: A binary search algorithm is used to adjust the upper and lower limits of the pipeline stage time, thereby finding the minimum value of the maximum running time in the pipeline stage, and obtaining the corresponding pipeline division manner and the corresponding parallel strategy, which comprise the following steps: Step B1, initialization of parameters: set the left boundary left and the right boundary right of the current pipeline stage as the first module, set the number of currently divided pipeline stages stages as 0, set the pipeline stage and the parallel strategy of all pipeline stages as empty sets, and set the parallel strategy of the current pipeline stage as empty; wherein the left boundary left and the right boundary right refer to the first module and the last module of the current pipeline stage; Step B2, judge whether left < 2N, if yes, go to step B3, if no, go to step B4; wherein 2N is the total number of modules; Step B3, set the left boundary left of the current pipeline stage as the first module of the current pipeline stage, set the right boundary right of the current pipeline stage as the last module of the current pipeline stage, set the parallel strategy of the current pipeline stage as the parallel strategy of the first module, and go to step B2; Step B3, by solving the knapsack problem, the optimal running time of the current pipeline stage and the corresponding parallel strategy are obtained by using the left boundary left, the right boundary right, the number of computing nodes of the current pipeline stage, and the preset memory limit M. Then, it is judged whether the constraint condition is met: right≤2N, and the running time of the current pipeline stage is less than or equal to the preset time limit Q. If yes, the right boundary right is added by 1, and the corresponding parallel strategy is saved to the current pipeline stage parallel strategy. If no, it is judged whether the current pipeline stage parallel strategy is empty. If empty, false is output, and the algorithm is ended. If not empty, (left, right - 1) is added to the pipeline stage, that is, it is returned to the last pipeline stage that meets the constraint condition, the first module of which is the left boundary left, and the last module is the right boundary right - 1. The parallel strategy corresponding to the last pipeline stage that meets the constraint condition is added to the parallel strategy of all pipeline stages. The right boundary right is assigned to the left boundary left, the number of currently divided pipeline stages is added by 1, and the current pipeline stage parallel strategy is set to empty. Then, step B2 is returned. Step B4, judging whether the current divided stages is less than or equal to the pipeline parallelism S, if yes, output true, otherwise, output false; wherein, the true output means that the pipeline division mode and the corresponding parallel strategy meet the limitation of the median ; the false output means that the pipeline division mode and the corresponding parallel strategy do not meet the limitation of the median .

7. The reasoning method of a dynamic routing hybrid expert model according to claim 1, wherein, The parallel reasoning of the dynamic routing hybrid expert model is controlled by using the searched pipeline division mode and the corresponding parallel strategy, which includes: The dynamic routing hybrid expert model is deployed in an inference framework, the inference framework is configured by using the searched pipeline division mode and the corresponding parallel strategy, and the parallel reasoning of the dynamic routing hybrid expert model is controlled by the inference framework.

8. An inference system of a dynamic routing hybrid expert model, characterized by, The method for implementing any one of claims 1-7 includes: A module sequence construction unit is configured to use modules contained in each transformer layer of the dynamic routing hybrid expert model as basic units to form a module sequence to be divided. The modules include attention modules and hybrid expert modules. A pipeline division scheme and parallel strategy searching unit is configured to search, in the module sequence to be divided, a pipeline division mode and a corresponding parallel strategy that correspond to the minimum value of the maximum running time of all pipeline stages under the preset memory limit, according to the running time of each module when different parallel strategies are used under different pipeline division modes, and in combination with the memory usage of each module. A parallel reasoning unit is configured to control the parallel reasoning of the dynamic routing hybrid expert model by using the searched pipeline division mode and the corresponding parallel strategy.

9. A processing device, characterized by One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1-7. The computer program is executed by the processor to implement the method of any one of claims 1-7.

10. A readable storage medium, storing a computer program, characterized in that, ​

Images