Mixed expert sparse large model inference method, system and electronic device

Abstract

The application provides a hybrid expert sparse large model inference method and system and electronic equipment, relates to the technical field of neural network inference optimization, and the method comprises the following steps: in the decoding stage of the hybrid expert sparse large model, in response to the decision data of parallel strategy switching, the strategy switching step is sequentially executed for each hybrid expert layer; the target weight is loaded to the switching buffer of the current layer, the original weight is continuously used for inference before the loading is completed, the target weight is enabled after the loading is completed, the space occupied by the original weight is used as the switching buffer of the next layer, and the remaining layers continue to infer according to the respective parallel strategies. The application is used for solving the problem that the inference service is interrupted and the memory overhead is large when the parallel strategy is switched in the decoding stage of the hybrid expert sparse large model, and can keep the inference service uninterrupted and not significantly increase the memory overhead in the parallel strategy switching process in the decoding stage of the hybrid expert sparse large model.

Description

Technical Field

[0001] This application relates to the field of neural network inference optimization technology, and in particular to hybrid expert sparse large model inference methods, systems and electronic devices. Background Technology

[0002] In recent years, the parameter size of large models has grown rapidly. Sparse activation architectures, represented by Mixture-of-Experts (MoE) sparse large models, have become an effective method for efficiently scaling large models. MoE sparse large models replace the multilayer perceptron layers in traditional dense models with a single gating function and multiple expert networks. Each term, after passing through the gating function, only activates a subset of experts for computation, thus achieving sublinear growth in computational cost with model capacity. The computational process of large model inference is divided into a pre-filling stage and a decoding stage. The pre-filling stage is computationally intensive, while the decoding stage is memory-intensive. Using the same computational resources in both stages may cause interference. To eliminate this interference, a pre-filling-decoding separation technique has been proposed, allowing the pre-filling and decoding stages to use different computing devices, enabling each stage to focus on its respective optimization objective. In distributed inference of Mixture-of-Experts (MoE) sparse large models, an expert parallelism model is typically adopted, evenly distributing all experts across different devices, while the multi-head attention layer uses a data parallelism approach.

[0003] During the inference process in the decoding phase, the load fluctuates dynamically with the number of user requests. When the batch size is large, the computational density of the decoding phase increases, and expert parallelism can achieve high throughput. When the batch size is small, the full-switching communication operation in expert parallelism becomes the main performance bottleneck, at which point tensor parallelism can effectively reduce latency. Currently, some technical solutions have been researched for switching parallel strategies in hybrid expert sparse large models. Some solutions determine the target parallel strategy from multiple candidate parallel strategies based on the load and switch accordingly. However, these solutions are geared towards training scenarios, and the switching is a synchronous operation between training cycles, making it difficult to directly apply to online inference scenarios with stringent requirements for service continuity. Other solutions implement runtime switching between expert parallelism and tensor parallelism during the inference phase. However, these solutions use a method of simultaneously resident both layouts in the accelerated processor memory and completely replacing all weight fragments during the switching process. This results in a long switching time and requires additional memory overhead to resident the weight copies of the two layouts.

[0004] However, the aforementioned existing technologies still have the following problems when applied to online inference services during the decoding phase of hybrid expert sparse large models: During parallel strategy switching, the simultaneous switching of all weights can cause each layer to wait for new weights to become available, making it difficult to maintain uninterrupted inference service; the simultaneous resident of corresponding weight copies in the accelerated processor's memory for multiple parallel strategies significantly increases memory overhead. Furthermore, the existing load balancing scheme, which consistently uses expert parallelism, cannot dynamically switch between expert parallelism and tensor parallelism based on the real-time load status during the decoding phase. In small-batch decoding scenarios, the full-switching communication overhead of expert parallelism leads to performance degradation. Summary of the Invention

[0005] In view of this, embodiments of this application provide a hybrid expert sparse large model inference method, system, and electronic device to eliminate or improve one or more defects existing in the prior art.

[0006] The first aspect of this application provides a hybrid expert sparse large model inference method, including: In the decoding phase of the hybrid expert sparse large model, in response to decision data indicating the switching of the current parallel strategy to the target parallel strategy, a strategy switching step is sequentially performed on each hybrid expert layer of the hybrid expert sparse large model to switch each hybrid expert layer to the target parallel strategy layer by layer. The strategy switching step includes: Load the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer; before the target weights are loaded, control the current hybrid expert layer to continue using its original weights to perform inference calculations; after the target weights are loaded, control the current hybrid expert layer to enable the target weights in the switching buffer, and use the space occupied by the original weights as the switching buffer of the next hybrid expert layer. In addition, control the remaining hybrid expert layers, other than the current hybrid expert layer, to continue performing inference computations according to their respective parallel strategies.

[0007] In some embodiments of this application, before sequentially performing the policy switching step on each hybrid expert layer of the hybrid expert sparse large model, the method further includes: Continuously acquire the load index values ​​of the hybrid expert sparse large model during the decoding stage; If all the load index values ​​obtained consecutively are greater than the first load threshold or less than the second load threshold, then decision data is generated to indicate whether to switch the current parallel strategy to the target parallel strategy corresponding to the first load threshold or the target parallel strategy corresponding to the second load threshold; wherein, the first load threshold is greater than the second load threshold.

[0008] In some embodiments of this application, the target parallel strategy corresponding to the first load threshold is an expert parallel strategy, and the target parallel strategy corresponding to the second load threshold is a tensor parallel strategy. If all the load index values ​​obtained consecutively are greater than the first load threshold or less than the second load threshold, then decision data is generated to indicate whether to switch the current parallel strategy to the target parallel strategy corresponding to the first load threshold or the target parallel strategy corresponding to the second load threshold, including: A finite state machine is used, and the state values ​​of the finite state machine are updated according to a preset state update rule; wherein, the preset state update rule includes: when the load index value is greater than the first load threshold, the state value is increased; when the load index value is less than the second load threshold, the state value is decreased. If the state value reaches the first boundary value and the current parallel strategy is the tensor parallel strategy, then decision data is generated to indicate switching the current parallel strategy to the expert parallel strategy. If the state value reaches the second boundary value and the current parallel strategy is the expert parallel strategy, then decision data is generated to indicate switching the current parallel strategy to the tensor parallel strategy; wherein the first boundary value is greater than the second boundary value.

[0009] In some embodiments of this application, the second boundary value is 0; the first boundary value is 2. n -1, where n is an integer greater than 1; Increasing the state value includes: increasing the state value by 1; Decreasing the state value includes reducing the state value by 1.

[0010] In some embodiments of this application, the continuous acquisition of load metric values ​​of the hybrid expert sparse large model during the decoding phase includes: In each decoding iteration of the decoding phase, the number of tokens generated by each in-process request in this decoding iteration is obtained, and the number of generated tokens is used as the load index value of this decoding iteration.

[0011] In some embodiments of this application, before sequentially performing the policy switching step on each hybrid expert layer of the hybrid expert sparse large model, the method further includes: Allocate a corresponding switching buffer for the first hybrid expert layer in the hybrid expert sparse large model to execute the policy switching step; The method further includes: After the policy switching step is completed in the last hybrid expert layer of the hybrid expert sparse large model, the switching buffer corresponding to the last hybrid expert layer is released.

[0012] In some embodiments of this application, the decoding phase of the hybrid expert sparse large model is performed by an inference plane and a switching plane; The inference plane is used to perform inference computations for the hybrid expert sparse large model; The switching plane is used to sequentially execute policy switching steps on each hybrid expert layer of the hybrid expert sparse large model.

[0013] In some embodiments of this application, the switching plane employs a synchronization mechanism to perform the operation of controlling the current hybrid expert layer to enable the target weight in the switching buffer, and the operation of updating the parallel policy flag of the current hybrid expert layer, so as to avoid conflicts with the inference computation of the inference plane; The synchronization mechanism includes at least one of the following: mutex, spinlock, atomic operation, and semaphore.

[0014] In some embodiments of this application, controlling the current hybrid expert layer to enable the target weight in the switching buffer includes: The pointers to the original weights of the current hybrid expert layer are swapped with the pointers to the target weights in the switching buffer, and the parallel policy flag of the current hybrid expert layer is updated accordingly.

[0015] In some embodiments of this application, loading the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer includes: The target weights are loaded from the central processing unit memory into the switching buffer of the current hybrid expert layer; wherein the switching buffer is located in the video memory of the accelerated processor used to perform inference computation of the hybrid expert sparse large model.

[0016] In some embodiments of this application, the hybrid expert sparse large model adopts an architecture that separates the pre-filling stage from the decoding stage; The calculations in the pre-filling stage and the calculations in the decoding stage are performed by different computing devices. The computing device used to perform the calculations in the pre-filling stage transmits the key-value cache generated in the pre-filling stage to the computing device used to perform the calculations in the decoding stage. The method further includes: In the decoding phase, when an expert parallel strategy is adopted, each hybrid expert layer of the hybrid expert sparse large model is executed layer by layer in each decoding iteration: attention output is obtained through multi-head attention calculation, expert number of each word activated in the current batch is obtained through gating function, and each word is routed to the computing device where the corresponding expert is located based on the expert number and expert calculation is performed to generate the next word; wherein, in the decoding phase, the current batch consists of words generated by each in-process request in this decoding iteration; The computing device used to perform the computation in the pre-filling stage is configured to perform the following before the decoding stage of the hybrid expert sparse large model: In the pre-filling stage of the hybrid expert sparse large model, receiving a sequence of prompt words input by the user, and sequentially performing multi-head attention computation on each hybrid expert layer of the hybrid expert sparse large model: obtaining the expert number activated by each lexical in the current batch through a gating function, and routing each lexical to the computing device where the corresponding expert is located based on the expert number and performing expert computation to generate the first lexical and produce a key-value cache; wherein, the hybrid expert layer of the hybrid expert sparse large model adopts an expert parallel strategy in the pre-filling stage, and in the pre-filling stage, the current batch is composed of the prompt word sequence.

[0017] In some embodiments of this application, after obtaining the expert number of each word activated in the current batch through a gating function, and before routing each word to the computing device where the corresponding expert is located based on the expert number and performing expert calculation, the method further includes: Based on the expert IDs activated for each lexical unit in the current batch, load balancing scheduling is performed on the lexical units to be processed on each computing device.

[0018] In some embodiments of this application, the load balancing scheduling of the unprocessed lexical units on each computing device based on the expert IDs activated for each lexical unit in the current batch includes: The expert IDs of all token activations in the current batch are obtained through a full collection operation; The number of lexical units to be processed by local experts on each of the aforementioned computing devices is counted, and the local experts are sorted according to the number of lexical units to obtain the sorting results; Based on the average token quota of each computing device and the sorting result, excess tokens on computing devices that exceed the average token quota are preferentially allocated to the computing device with the largest free quota in order to minimize the number of missing experts after load balancing; wherein, the missing expert is an expert that needs to be processed on a computing device but is not stored locally on that computing device.

[0019] In some embodiments of this application, after load balancing scheduling of the lexical units to be processed on each of the computing devices, and before routing each lexical unit to the computing device where the corresponding expert resides based on the expert number and performing expert computation, the method further includes: If a missing expert still exists after load balancing, the expert parameters of the missing expert are loaded from the CPU memory to the video memory of the corresponding computing device, and the loaded expert parameters are released after the word calculation involving the missing expert in the current batch is completed; wherein, the expert that needs to be processed on the computing device and has been stored locally is the hit expert.

[0020] In some embodiments of this application, the process of loading the expert parameters of the missing expert from the central processing unit memory to the video memory of the corresponding computing device overlaps with the communication operation of routing each word to the computing device where the corresponding expert is located based on the expert number, and the calculation operation on the computing device for the hit expert.

[0021] In some embodiments of this application, after performing expert calculations, the method further includes: Each lexical is routed back to the computing device where it was located before the gating function was executed via a fully switched communication operation for multi-head attention computation of the next hybrid expert layer.

[0022] The second aspect of this application provides a decoding stage computing device for implementing the decoding stage computation in the hybrid expert sparse large model inference method described in the first aspect above, comprising: The strategy switching module is used to, in response to decision data indicating the switching of the current parallel strategy to the target parallel strategy, sequentially execute strategy switching steps on each hybrid expert layer of the hybrid expert sparse large model during the decoding phase of the hybrid expert sparse large model, so as to switch each hybrid expert layer to the target parallel strategy layer by layer. The strategy switching step includes: loading the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer; before the target weights are loaded, controlling the current hybrid expert layer to continue using its original weights to perform inference calculations; after the target weights are loaded, controlling the current hybrid expert layer to enable the target weights in the switching buffer, and using the space occupied by the original weights as the switching buffer of the next hybrid expert layer. In addition, control the remaining hybrid expert layers, other than the current hybrid expert layer, to continue performing inference computations according to their respective parallel strategies.

[0023] A third aspect of this application provides a pre-filling stage computing apparatus for performing pre-filling stage computations in the hybrid expert sparse large model inference method described in the first aspect above, comprising: The inference computation module is used in the pre-filling stage of the hybrid expert sparse large model to receive the prompt word sequence input by the user, and to perform the following sequentially layer by layer on each hybrid expert layer of the hybrid expert sparse large model: multi-head attention computation, obtaining the expert number activated by each word in the current batch through a gating function, routing each word to the computing device where the corresponding expert is located based on the expert number and performing expert computation to generate the first word and generate a key-value cache; wherein, the hybrid expert layer of the hybrid expert sparse large model adopts an expert parallel strategy in the pre-filling stage, and in the pre-filling stage, the current batch is composed of the prompt word sequence; The transmission module is used to transmit the key-value cache generated in the pre-filling stage to the computing device in the decoding stage.

[0024] A fourth aspect of this application provides a hybrid expert sparse large model inference system, comprising: a decoding stage computing device, the decoding stage computing device including an inference plane and a switching plane for jointly executing the hybrid expert sparse large model inference method described in the first aspect above; The inference plane is used to perform inference computations for the hybrid expert sparse large model; The switching plane is used to sequentially execute policy switching steps on each hybrid expert layer of the hybrid expert sparse large model.

[0025] In some embodiments, the hybrid expert sparse large model inference system further includes a pre-filling stage computing device for performing the pre-filling stage computation in the hybrid expert sparse large model inference method described in the first aspect above.

[0026] A fifth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the hybrid expert sparse large model inference method.

[0027] The sixth aspect of this application provides another electronic device, including a memory, multiple processors, and a computer program stored in the memory and executable on the multiple processors. When the multiple processors execute the computer program, one processor performs operations on the inference plane, and another processor performs operations on the switching plane, jointly implementing the hybrid expert sparse large model inference method described in the first aspect above.

[0028] The seventh aspect of this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the hybrid expert sparse large model inference method described in the first aspect above.

[0029] The eighth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the hybrid expert sparse large model inference method described in the first aspect above.

[0030] The hybrid expert sparse large model inference method provided in this application, in response to switching decision data during the decoding phase, sequentially executes policy switching steps for each hybrid expert layer, enabling asynchronous switching layer by layer. In the policy switching step, the target weights required for the target parallel policy are loaded into the switching buffer of the current hybrid expert layer. Before loading is complete, the current hybrid expert layer continues to perform inference calculations using its original weights. After loading is complete, the current hybrid expert layer activates the target weights in the switching buffer and uses the space occupied by the original weights as the switching buffer for the next hybrid expert layer. Simultaneously, it controls the other hybrid expert layers, except the current one, to continue performing inference calculations according to their respective parallel policies. Through the aforementioned layer-by-layer switching mechanism, inference services remain uninterrupted during parallel strategy switching. The space occupied by old weights is reused layer by layer, avoiding the simultaneous presence of corresponding weight copies in the accelerator's memory for multiple parallel strategies, thus reducing memory overhead. The current hybrid expert layer continues to use the original weights to perform inference calculations during target weight loading, while other hybrid expert layers continue to perform inference calculations according to their respective parallel strategies. This allows the switching operation and inference calculations to be executed asynchronously and in parallel, and the time overhead of the switching operation can be masked by the inference calculation process, reducing the impact of switching on inference latency. The space occupied by old weights is directly reused as a switching buffer for the next hybrid expert layer, eliminating the need to allocate new buffer space for each layer. This reduces peak memory usage and the frequency of buffer allocation and release operations, improving memory management efficiency. Because a layer-by-layer rather than simultaneous overall switching approach is adopted, the instantaneous resource requirements for a single-layer switch are smaller, avoiding the concentrated occupation of accelerator computing resources and memory bandwidth by an overall switch. This layer-by-layer switching mechanism does not depend on a specific parallel strategy type and can be applied to switching between different parallel strategies, improving the adaptability of the inference system under different load scenarios.

[0031] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0032] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0033] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a schematic diagram of the basic architecture of a hybrid expert sparse large model.

[0034] Figure 2 This is a schematic diagram of the first process of the hybrid expert sparse large model inference method in one embodiment of this application.

[0035] Figure 3 This is a schematic diagram of the second process of the hybrid expert sparse large model inference method in one embodiment of this application.

[0036] Figure 4 This is a schematic diagram of the third process of the hybrid expert sparse large model inference method in one embodiment of this application.

[0037] Figure 5 This is a schematic diagram of the architecture of the inference plane and switching plane in the decoding stage computing device according to an embodiment of this application.

[0038] Figure 6 This is a schematic diagram of the architecture of a hybrid expert sparse large model inference system in one embodiment of this application.

[0039] Figure 7 This is a schematic diagram of the basic calculation process performed by the computing device in the pre-filling stage according to an embodiment of this application.

[0040] Figure 8 This is a schematic diagram illustrating the specific steps of load balancing scheduling and on-demand loading of missing experts in one embodiment of this application.

[0041] Figure 9 This is a schematic diagram illustrating an example of the execution of a load balancing scheduling process in one embodiment of this application.

[0042] Figure 10 This is a schematic diagram of the overall execution flow of the load balancing optimization method for hybrid expert sparse large model split parallel inference in an application example of this application.

[0043] Figure 11 This is a pseudocode example of the inference plane execution algorithm in an application example of this application.

[0044] Figure 12 This is a pseudocode example of the switching plane execution algorithm in an application example of this application.

[0045] Figure 13 This is a schematic diagram illustrating an application example of switching from tensor parallelism to expert parallelism asynchronously layer by layer in this application.

[0046] Figure 14 This is a schematic diagram illustrating an application example of switching from expert parallelism to tensor parallelism in an asynchronous, layer-by-layer manner.

[0047] Figure 15(a) is a schematic diagram showing the comparison results of output lexical delay in an application example of this application.

[0048] Figure 15(b) is a schematic diagram showing the comparison results of the first word delay in an application example of this application.

[0049] Figure 15(c) is a schematic diagram showing the comparison results of system throughput in an application example of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0051] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0052] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0053] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0054] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0055] First, it should be noted that... (See also) Figure 1 The basic architecture of the hybrid expert sparse large model includes a multi-head attention layer and a hybrid expert layer. This architecture can be repeated multiple times and chained together to form a multi-layer deep neural network. Compared to the traditional dense large model, the hybrid expert sparse large model replaces the multi-layer perceptron (MLP) layer in the traditional dense model with a gating function and multiple experts, FFN0, FFN1, and FFN2.E-1 The parameters of the hybrid expert layer model for different experts are represented by ; FFN (Feed-Forward Network) represents the feedforward network; and the multi-head attention layer is the same as in traditional dense models. In the hybrid expert sparse large model, each token, after passing through a gating function, has its affinity score calculated for each expert. Then, the TopK experts with the highest affinity scores are selected for computation. Therefore, in the hybrid expert sparse large model, each token only activates some experts for computation, and each token does not need to be computed by all experts, thus achieving sublinear growth of computational cost with model capacity and achieving rapid expansion of model parameters. Compared with traditional dense large models, the hybrid expert sparse large model can significantly reduce computational cost with the same number of parameters. In other words, the hybrid expert sparse large model can effectively reduce the training cost of large models, achieving sublinear growth of computational cost with model capacity. Currently, most mainstream large models adopt the MoE hybrid expert architecture to reduce costs.

[0056] The computational process of large-scale model inference can be divided into two stages: the pre-filling stage and the decoding stage. The pre-filling stage refers to the computational process of generating the first lexical unit by inputting prompts (such as user questions) into the model. The decoding stage refers to the computational process of iteratively generating the next lexical unit based on the current input sequence and the latest generated lexical unit after generating the first lexical unit. Decoding is an iterative computational process, generating a new lexical unit in each iteration until the maximum generation length is reached or a terminating lexical unit is encountered. The decoding stage uses KV-Cache technology to cache previously computed key and value vectors, avoiding repeated computation of already processed inputs when generating new lexical units, thus significantly saving computational power, reducing latency, and improving throughput. The pre-filling stage is computationally intensive, while the decoding stage is memory-intensive; the two stages using the same computational resources may interfere with each other. Therefore, some studies have proposed a pre-filling-decoding PD separation technique, which separates the pre-filling and decoding stages using different computing devices to eliminate interference between the two stages. This allows each stage to focus on its respective optimization objective: the pre-filling stage focuses on optimizing the latency of the first word, while the decoding stage focuses on optimizing the latency of the output word. In the PD separation technique, the key-value cache (KV-cache) calculated in the pre-filling stage needs to be transmitted to the decoding device before it can perform subsequent calculations. Currently, PD separation has become one of the mainstream methods for parallel inference services of large models. Among them, the KV-Cache (Key-Value Cache) in large models is a core optimization technique used to accelerate the inference process of Transformer models. Its principle is to cache previously calculated key and value vectors, avoiding repeated calculations of already processed inputs when generating new words, thereby significantly saving computing power, reducing latency, and improving throughput.

[0057] In the autoregressive generation process of Large Language Models (LLM), each step requires calculating the attention weights between the current word and all previous words. KV-Cache stores the key and value vectors of these historical words, allowing the model to calculate the attention between the current word and the historical words in the cache when generating new words, without having to recalculate the entire sequence.

[0058] During the inference process in the decoding phase, the load fluctuates dynamically with the number of user requests. When the batch size is large, the computational density of the decoding phase increases, and expert parallelism can achieve high throughput. When the batch size is small, the full-switching communication operation in expert parallelism becomes the main performance bottleneck, at which point tensor parallelism can effectively reduce latency. However, when switching between expert parallelism and tensor parallelism strategies, the weights required by the target parallel strategy need to be loaded into the accelerator's memory, and the loading process takes time. If all weights are switched simultaneously, each layer may need to wait for the new weights to be ready during the switch, causing the inference service to pause and affecting service quality. In addition, if multiple parallel strategies have corresponding weight copies resident in the accelerator's memory at the same time, it will significantly increase memory overhead, which is unbearable in large-scale model deployment scenarios. Currently, some technical solutions have implemented runtime switching between expert parallelism and tensor parallelism in the inference phase, but they use the method of having both layouts resident in the accelerator's memory at the same time and replacing all weight fragments at the same time. The switching time is long, and additional memory overhead is required to resident weight copies of the two layouts. Other solutions consistently employ expert parallelism for load balancing, failing to dynamically switch between expert parallelism and tensor parallelism based on real-time load conditions. In small-batch decoding scenarios, the full-switching communication overhead of expert parallelism leads to performance degradation. Furthermore, while some solutions can adjust the expert layout, this requires pausing the inference service during the adjustment process, resulting in significant redistribution overhead.

[0059] Based on this, in order to solve the problems of inference service interruption and high memory overhead during parallel strategy switching in the decoding stage of hybrid expert sparse large models, embodiments of this application provide a hybrid expert sparse large model inference method, a hybrid expert sparse large model inference device for executing the hybrid expert sparse large model inference method, a hybrid expert sparse large model inference system, an electronic device, a computer-readable storage medium, and a computer program product, which can maintain uninterrupted inference service and not significantly increase memory overhead during parallel strategy switching in the decoding stage of hybrid expert sparse large models.

[0060] The following examples will provide a detailed description.

[0061] Based on this, embodiments of this application provide a hybrid expert sparse large model inference method that can be executed by the decoding stage computing device in a hybrid expert sparse large model inference system. See [link to relevant documentation]. Figure 2 The hybrid expert sparse large model inference method specifically includes the following: Step 100: In the decoding phase of the hybrid expert sparse large model, in response to decision data indicating the switching of the current parallel strategy to the target parallel strategy, a strategy switching step is sequentially performed on each hybrid expert layer of the hybrid expert sparse large model to switch each hybrid expert layer to the target parallel strategy layer by layer.

[0062] It is understood that the decoding stage refers to the computational process by which the hybrid expert sparse large model iteratively generates the next word based on the current input sequence and the latest generated word after generating the first word.

[0063] The decision data refers to the instructions or signals used to trigger the switching of parallel strategies. The decision data can be generated in various ways. As one example, the decision data can be generated based on the load status of the decoding phase. For instance, when the batch size of multiple consecutive decoding batches is detected to be greater than a first load threshold, decision data indicating a switch to an expert parallel strategy is generated; when the batch size of multiple consecutive decoding batches is less than a second load threshold, decision data indicating a switch to a tensor parallel strategy is generated. As another example, the decision data can originate from switching instructions from an external system, such as switching instructions manually triggered by the user, switching instructions issued by the task scheduling system based on overall resource status, or switching instructions issued by the monitoring system based on preset rules. As yet another example, the decision data can also be generated based on a timed switching strategy, such as using an expert parallel strategy during a preset time period and a tensor parallel strategy during other time periods. Regardless of how the decision data is generated, once step 100 obtains the decision data, the strategy switching steps are executed sequentially for each hybrid expert layer. The execution of the strategy switching steps defined in step 100 does not depend on the specific generation method of the decision data.

[0064] The target parallel strategy refers to the parallel strategy to be adopted by the hybrid expert layer after the switch is completed. It is obvious that the target parallel strategy will be different from the current parallel strategy. For example, the current parallel strategy may be a tensor parallel strategy, and the target parallel strategy may be an expert parallel strategy; or, the current parallel strategy may be an expert parallel strategy, and the target parallel strategy may be a tensor parallel strategy. The specific type of the target parallel strategy is indicated by the decision data. It is understood that the specific type of the target parallel strategy is not limited to the two mentioned above; for example, it may also include other parallel strategies such as data parallel strategies and pipeline parallel strategies, which can be set according to actual application requirements.

[0065] Furthermore, each hybrid expert layer refers to a neural network layer in the hybrid expert sparse large model that contains multiple expert networks. During step 100, the policy switching step is performed on each hybrid expert layer sequentially; that is, the first hybrid expert layer is switched first, then the second, and so on, until all hybrid expert layers have been switched. Since each hybrid expert layer can continue inference according to the target parallel policy after the switch is completed, while other hybrid expert layers that have not yet been switched can continue inference according to their current parallel policy, the entire inference service will not be paused due to the parallel policy switch. A detailed explanation of the policy switching step is as follows: See Figure 2 The strategy switching step in step 100 specifically includes the following: Step 110: Load the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer; before the target weights are loaded, control the current hybrid expert layer to continue using its original weights to perform inference calculations; after the target weights are loaded, control the current hybrid expert layer to enable the target weights in the switching buffer, and use the space occupied by the original weights as the switching buffer of the next hybrid expert layer.

[0066] And, step 120: control the remaining hybrid expert layers, except for the current hybrid expert layer, to continue performing inference computation according to their respective parallel strategies.

[0067] It is understood that the target weights refer to the model weight parameters required for each expert network in the current hybrid expert layer when the target parallel strategy is adopted. When the target parallel strategy is an expert parallel strategy, the expert networks deployed on each computing device may differ from those deployed under the current parallel strategy; when the target parallel strategy is a tensor parallel strategy, the complete expert network weights need to be deployed on each computing device.

[0068] The target weights can be loaded from various sources. As one example, the target weights can be loaded from the central processing unit (CPU) memory into the switching buffer, which is located in the video memory of an accelerator processor (such as a graphics processing unit, GPU) used for performing inference computation. As another example, the target weights can also be loaded from the video memory of other accelerator processors via a network or bus transfer into the switching buffer of the current accelerator processor. As yet another example, the target weights can also be loaded from an external storage device (such as a solid-state drive).

[0069] The switching buffer refers to a temporary storage space allocated in the accelerator processor's video memory to store the target weights. The timing of the switching buffer allocation can vary. For example, it can be allocated uniformly before the first hybrid expert layer executes the policy switching step, with subsequent layers reusing the space occupied by the original weights; alternatively, it can be allocated separately for each hybrid expert layer before the switching is performed and released separately after the switching is completed.

[0070] The loading process refers to the transfer of target weight data to the switching buffer. During loading, the target weights are gradually transferred to the switching buffer. Loading completion refers to the moment when all target weight data has been transferred and is available for use by the current hybrid expert layer.

[0071] The original weights refer to the model weight parameters used by the current hybrid expert layer before the switch, i.e., the weights of each expert network in the hybrid expert layer under the current parallel strategy. During the loading of the target weights, the current hybrid expert layer continues to perform inference computation using the original weights and will not suspend service due to the loading of the target weights. The control mentioned in step 110 refers to the execution scheduling mechanism of the inference system maintaining the state of the current hybrid expert layer using the original weights for inference computation without interrupting its inference process.

[0072] After loading, the current hybrid expert layer uses the target weights in the switching buffer to replace the original weights for subsequent inference computations. There are several ways to enable this. As one example, it can be implemented by pointer swapping: exchanging the pointer to the original weights with the pointer to the target weights in the switching buffer, and updating the parallel policy flag of the current hybrid expert layer so that subsequent inference computations of this layer use the target weights and the new parallel policy. Another example is by modifying the weight index table or memory address mapping to update the weight addresses referenced by the current hybrid expert layer to the address of the target weights.

[0073] After the target weights are loaded and enabled, the memory space occupied by the original weights is no longer needed by the current layer. Using this memory space as a switching buffer for the next hybrid expert layer can be implemented in several ways. As one example, a pointer to the original weight space can be directly passed to the next hybrid expert layer, allowing it to load the target weights into that space. Another example is to record the memory address of the original weight space in a global buffer management table. When the next hybrid expert layer needs to switch buffers, it retrieves the available memory address from this table and uses it directly. This cycle can reduce peak memory usage.

[0074] The remaining hybrid expert layers refer to all other hybrid expert layers in the hybrid expert sparse large model except for the hybrid expert layer currently performing a policy switching step. These layers include hybrid expert layers that have completed the switching and those that have not yet performed the switching. Layers that have completed the switching perform inference computation according to the target parallel policy, while layers that have not yet switched perform inference computation according to their current parallel policy. The control described in step 120 refers to the execution scheduling mechanism of the inference system maintaining these layers independently and continuing to perform inference computation according to their current parallel policy, without being paused or blocked due to the policy switching of the current layer. As an example, the policy switching operation and the inference computation operation can be managed separately through independent execution threads or execution streams, with both executed asynchronously in parallel. As another example, inference computation tasks can be continuously submitted to the accelerator processor for execution through event-driven or task queue methods, while the policy switching operation is completed independently in the background without blocking the normal scheduling of inference computation tasks.

[0075] As described above, the hybrid expert sparse large model inference method provided in this application, in response to switching decision data during the decoding phase, sequentially executes policy switching steps for each hybrid expert layer, achieving asynchronous switching layer by layer. In the policy switching step, the target weights required by the target parallel policy are loaded into the switching buffer of the current hybrid expert layer. Before loading is complete, the current hybrid expert layer continues to use its original weights to perform inference calculations. After loading is complete, the current hybrid expert layer activates the target weights in the switching buffer and uses the space occupied by the original weights as the switching buffer for the next hybrid expert layer. Simultaneously, it controls the remaining hybrid expert layers (excluding the current one) to continue performing inference calculations according to their respective parallel policies. The parallel policy switching during the decoding phase is implemented asynchronously. The inference service runs normally during the switching process, without causing service interruption, resulting in low switching overhead. Through the above layer-by-layer switching mechanism, the inference service remains uninterrupted during parallel policy switching, and the space occupied by the old weights is reused layer by layer. This avoids the simultaneous presence of corresponding weight copies in the accelerated processor's memory for multiple parallel policies, reducing memory overhead.

[0076] To further avoid frequent switching of parallel strategies due to instantaneous fluctuations in load during the decoding phase and improve system stability, a hybrid expert sparse large model inference method is provided in this application embodiment. (See also...) Figure 3 The hybrid expert sparse large model inference method, which can be executed by the decoding stage computing device, further includes the following steps prior to step 100: Step 010: Continuously acquire the load index values ​​of the hybrid expert sparse large model during the decoding stage.

[0077] It should be noted that the load metric value refers to a quantitative value used to measure the current load state of the hybrid expert sparse large model during the decoding phase. The load metric value can be of various types. As an example, the load metric value can be the batch size of the decoding batch, that is, the total number of tokens generated by each request in progress during the current decoding iteration. A larger batch size indicates a higher computational load in the current decoding phase. As another example, the load metric value can also be an indicator that reflects the system load state, such as request queue length, memory usage, computation unit utilization, or token throughput. The load metric value is obtained continuously; that is, the load metric value of the current batch is obtained once for each decoding iteration, thereby achieving real-time tracking of the load state during the decoding phase.

[0078] Step 020: If all the load index values ​​obtained consecutively are greater than the first load threshold or are less than the second load threshold, then generate decision data to indicate whether to switch the current parallel strategy to the target parallel strategy corresponding to the first load threshold or the target parallel strategy corresponding to the second load threshold; wherein, the first load threshold is greater than the second load threshold.

[0079] In step 020, the first load threshold and the second load threshold are preset values ​​used to determine the load status during the decoding stage. The first load threshold is greater than the second load threshold, meaning the first load threshold corresponds to a higher load level, and the second load threshold corresponds to a lower load level. For example, when the load index value is batch size, the first load threshold can be set to 16, and the second load threshold can be set to 4.

[0080] The term "continuous multiple acquisitions" refers to making decisions based on results from multiple consecutive batches, rather than relying solely on a single load metric value. As an example, a counter or queue can be used to record the comparison results of the most recent load metric values ​​with a threshold. Decision data is generated only when the results of multiple consecutive acquisitions all meet the same condition. For instance, if the load metric values ​​acquired three times consecutively are greater than a first load threshold, it is determined that the current decoding stage is in a state of sustained high load, and decision data is generated indicating that the current parallel strategy should be switched to the target parallel strategy corresponding to the first load threshold. Conversely, if the load metric values ​​acquired three times consecutively are less than a second load threshold, it is determined that the current decoding stage is in a state of sustained low load, and decision data is generated indicating that the current parallel strategy should be switched to the target parallel strategy corresponding to the second load threshold.

[0081] The aforementioned anti-jitter mechanism, which compares load index values ​​with thresholds multiple times consecutively, avoids frequent parallel strategy switching caused by instantaneous fluctuations in batch size. Switching decisions are only made when the load state continuously shifts in a certain direction, thus improving system stability.

[0082] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application embodiment can filter out instantaneous fluctuations in load status and trigger switching decisions only when the load status continuously meets the conditions, thereby reducing unnecessary parallel strategy switching operations and lowering switching overhead.

[0083] To achieve bidirectional adjustable load state tracking with extremely low storage overhead within a debouncing decision framework involving multiple consecutive threshold comparisons, and to prevent false triggering caused by instantaneous bidirectional load fluctuations, a hybrid expert sparse large model inference method provided in this application embodiment uses an expert parallel strategy for the target parallel strategy corresponding to the first load threshold and a tensor parallel strategy for the target parallel strategy corresponding to the second load threshold. Correspondingly, see [link to relevant documentation]. Figure 4 Step 020 of the hybrid expert sparse large model inference method, which can be executed by the decoding stage computing device, further includes the following: Step 021: Using a finite state machine, update the state value of the finite state machine according to a preset state update rule; wherein, the preset state update rule includes: when the load index value is greater than the first load threshold, increase the state value; when the load index value is less than the second load threshold, decrease the state value.

[0084] It is understood that the finite state machine refers to a mathematical model with a finite number of state values, capable of transitioning between states based on input conditions. The state values ​​are variables within the finite state machine used to record the load's duration. Each state value has a range, with the lower limit being the second boundary value and the upper limit being the first boundary value, where the first boundary value is greater than the second boundary value. After each load index value is acquired, the state value is updated based on a comparison between the load index value and a first load threshold and a second load threshold. When the load index value is greater than the first load threshold, the state value increases towards the first boundary value; when the load index value is less than the second load threshold, the state value decreases towards the second boundary value. When the load index value is between the first and second load thresholds, the state value remains unchanged. Through this bidirectional adjustment mechanism, the state value can reflect the continuous trend of the load state during the decoding phase; a single fluctuation will only cause a small change in the state value without immediately triggering a switch.

[0085] Step 022: If the state value reaches the first boundary value and the current parallel strategy is the tensor parallel strategy, then generate decision data to indicate switching the current parallel strategy to the expert parallel strategy.

[0086] Step 023: If the state value reaches the second boundary value and the current parallel strategy is the expert parallel strategy, then generate decision data to indicate switching the current parallel strategy to the tensor parallel strategy; wherein the first boundary value is greater than the second boundary value.

[0087] Specifically, the first boundary value corresponds to the trigger boundary on the high-load side, and the second boundary value corresponds to the trigger boundary on the low-load side. Reaching the first boundary value indicates that the decoding phase has been continuously under high load. If the current parallel strategy is a tensor parallel strategy, decision data is generated to switch the current parallel strategy to an expert parallel strategy to achieve higher throughput. Reaching the second boundary value indicates that the decoding phase has been continuously under low load. If the current parallel strategy is an expert parallel strategy, decision data is generated to switch the current parallel strategy to a tensor parallel strategy to achieve lower latency. While generating decision data, the state value can remain unchanged or be reset to its initial value according to preset rules, which can be set according to actual application requirements. By generating decision data only when the state value accumulates to the boundary value, it ensures that the switching decision is based on the continuous trend of the load state rather than instantaneous fluctuations, avoiding false triggers caused by instantaneous bidirectional load fluctuations.

[0088] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application uses a finite state machine and only needs to maintain one state value variable to track the continuous trend of the load, with low storage overhead; the bidirectional adjustment mechanism of the state value can distinguish between the instantaneous fluctuations and continuous trends of the load, and only triggers the switch when the load state continuously accumulates to the boundary value, which can reduce false triggering.

[0089] To further avoid frequent switching caused by instantaneous load fluctuations while flexibly adjusting the response speed to real load changes, in a hybrid expert sparse large model inference method provided in this application embodiment, the second boundary value is 0; the first boundary value is 2. n -1, where n is an integer greater than 1.

[0090] Specifically, the second boundary value corresponds to the trigger boundary on the low-load side and is 0; the first boundary value corresponds to the trigger boundary on the high-load side and is 2. n -1. The value of the state is in the range of 0 to 2. n -1, out of 2 nThere are several states. Here, n is an integer greater than 1, and its specific value can be set according to the actual application scenario. The larger the value of n, the wider the range of state values, requiring more consecutive unidirectional adjustments to reach the boundary value from the middle position, resulting in stronger anti-jitter performance but slower response to real load changes. Conversely, the smaller the value of n, the faster the response but relatively weaker the anti-jitter effect. For example, when n is 2, the state value range is 0 to 3, and the state value starts from the middle position and reaches the boundary value after 1 to 2 consecutive unidirectional adjustments; when n is 3, the state value range is 0 to 7, requiring more consecutive unidirectional adjustments to trigger a switch. The specific value of n can be determined based on empirical data or performance measurement results.

[0091] Based on this, the specific implementation method of increasing the state value in step 021 can be: increasing the state value by 1; the specific implementation method of decreasing the state value in step 021 can be: decreasing the state value by 1.

[0092] The finite state machine adjusts its state values ​​in steps of 1. When the load index value exceeds a first load threshold, the state value increases by 1, but not exceeding a first boundary value of 2. n -1; When the load index value is less than the second load threshold, the state value decreases by 1, but does not fall below the second boundary value of 0. This design with an adjustment step size of 1 each time allows the state value to smoothly reflect the continuous change trend of the load. A single fluctuation will only cause a small change in the state value and will not immediately trigger a switch.

[0093] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application fixes the state value step size to increase or decrease by 1 each time, and fixes the boundary values ​​to 0 and 2. n -1 allows for flexible control of the balance between stabilization strength and response speed by adjusting parameter n, with storage overhead being only a single state value variable, achieving extremely low-overhead stabilization switching decisions.

[0094] To further ensure that load assessment accurately reflects computational pressure while facilitating real-time acquisition, a hybrid expert sparse large model inference method is provided in this application embodiment, see [link to relevant documentation]. Figure 4 Step 010 of the hybrid expert sparse large model inference method, which can be executed by the decoding stage computing device, specifically includes the following: Step 011: In each decoding iteration of the decoding phase, obtain the number of tokens generated by each in-process request in this decoding iteration, and use the number of generated tokens as the load index value of this decoding iteration.

[0095] Specifically, the "in-progress request" refers to a user request in the decoding phase that has not yet generated a terminating token and has not reached the maximum generation length. In each decoding iteration, each in-progress request generates a new token. The total number of tokens generated by all in-progress requests in this decoding iteration is the batch size of this decoding iteration. Using the batch size as the load metric for this decoding iteration directly reflects the current computational and memory access pressure in the decoding phase. That is, the larger the batch, the more computationally intensive, and an expert parallel strategy is suitable for achieving higher throughput; the smaller the batch, the more prominent the memory access bottleneck, and a tensor parallel strategy is suitable for achieving lower latency. Using the number of generated tokens as the load metric eliminates the need for an additional performance monitoring module and can be obtained directly from the inference flow in the decoding phase in real time, facilitating lightweight load awareness.

[0096] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application uses the number of generated tokens in the decoding iteration as the load index value, which can directly reflect the current computation and memory access pressure in the decoding stage, and does not require an additional monitoring module, making the acquisition method simple and efficient.

[0097] To further avoid the additional overhead introduced by separately allocating and releasing buffers for each layer, a hybrid expert sparse large model inference method provided in this application embodiment is described below. Figure 3 The hybrid expert sparse large model inference method, which can be executed by the decoding stage computing device, further includes the following steps prior to step 100: Step 030: Allocate a corresponding switching buffer for the first hybrid expert layer in the hybrid expert sparse large model to execute the policy switching step.

[0098] The first hybrid expert layer to execute the policy switching step refers to the first hybrid expert layer processed according to the sequential execution order of the policy switching steps. Before executing the policy switching step for this first hybrid expert layer, a switching buffer is allocated for it. The allocation location of the switching buffer can be located in the GPU memory used for inference computation, and the allocated space is large enough to accommodate the target weight data required by a single hybrid expert layer. Since subsequent hybrid expert layers reuse the space occupied by the original weights released by the previous layer as their respective switching buffers when executing the policy switching step, the buffer allocation operation only needs to be performed once in this step during the entire layer-by-layer switching process, without the need to allocate a buffer separately for each hybrid expert layer, thereby reducing the frequency of GPU memory allocation operations.

[0099] Correspondingly, after step 100 of the hybrid expert sparse large model inference method, which can be executed by the decoding stage computing device, the following content is also specifically included: Step 200: After the policy switching step is completed in the last hybrid expert layer of the hybrid expert sparse large model, release the switching buffer corresponding to the last hybrid expert layer.

[0100] The last hybrid expert layer refers to the last hybrid expert layer processed according to the sequential execution of the policy switching steps. After the last hybrid expert layer completes the policy switching step, that is, after the target weight is enabled and the original weight space is released, its corresponding switching buffer is released. Since the original weight space of the current layer is reused as the switching buffer of the next layer after each layer switching is completed, the switching buffer held by the last hybrid expert layer after completing the switching is the remaining buffer space after layer-by-layer transfer and reuse. After releasing this buffer, the extra video memory space occupied by the entire parallel policy switching process is completely reclaimed. As an example, the release operation can return the buffer space to the video memory pool through the video memory management interface for use by other computing tasks. Thus, by using the above-mentioned buffer lifecycle management method of allocating once before the first layer, releasing once after the last layer, and cyclically reusing the intermediate layers, the additional overhead introduced by allocating and releasing buffers separately for each layer can be avoided, reducing peak video memory usage and reducing video memory fragmentation caused by frequent allocation and release operations.

[0101] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application can reduce the frequency of video memory allocation and release operations, reduce peak video memory usage, and improve video memory management efficiency by allocating and releasing the buffer only once and reusing the buffer in the middle layers in a cyclic manner.

[0102] To further ensure that inference computation and policy switching do not interfere with each other during layer-by-layer switching at the architectural level, and to further improve the reliability of uninterrupted inference service, a hybrid expert sparse large model inference method is provided in this application embodiment, see [link to relevant documentation]. Figure 5 In the hybrid expert sparse large model inference method that can be executed by the decoding stage computing device, the decoding stage of the hybrid expert sparse large model is executed by inference plane 01 and switching plane 02; the inference plane 01 is used to perform inference computation of the hybrid expert sparse large model; the switching plane 02 is used to sequentially execute policy switching steps for each hybrid expert layer of the hybrid expert sparse large model.

[0103] It is understood that the inference plane and the switching plane refer to two independent execution flows that are isolated, asynchronous, and executed in parallel during the decoding phase. The inference plane is specifically responsible for performing the inference computation of the hybrid expert sparse large model, that is, performing multi-head attention computation and expert computation according to the current parallel strategy of each hybrid expert layer, and does not participate in the parallel strategy switching. The switching plane runs independently in the background and is specifically responsible for performing the strategy switching steps sequentially for each hybrid expert layer. Since the two are isolated from each other, the switching operation of the switching plane will not block the inference computation of the inference plane, and the inference plane only needs to execute the corresponding inference service according to the current parallel strategy of each hybrid expert layer.

[0104] As one example, the inference plane and the switching plane can be two independent execution threads or processes, running on the same accelerator processor and interacting with each other via shared memory. As another example, the inference plane and the switching plane can run on different accelerator processors, communicating and coordinating via message passing or remote procedure calls. Regardless of the implementation method, the isolation between the two ensures that inference computation and policy switching do not interfere with each other, and that the inference service is not interrupted during the switching process.

[0105] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application divides the decoding stage into mutually isolated inference planes and switching planes. Inference computation and policy switching can be executed asynchronously and in parallel. The operation of the switching plane will not block the inference service of the inference plane. The continuity and stability of the inference service during the switching process are guaranteed at the architecture level.

[0106] To further prevent the inference plane from simultaneously reading the weight data or parallel policy flags of the same hybrid expert layer during the switching process, resulting in inconsistent data used for inference computation, based on the above embodiments, in the hybrid expert sparse large model inference method provided in this application embodiment, the switching plane uses a synchronization mechanism to perform the operation of controlling the current hybrid expert layer to enable the target weights in the switching buffer, and the operation of updating the parallel policy flags of the current hybrid expert layer, so as to avoid conflicts with the inference computation of the inference plane; wherein, the synchronization mechanism includes at least one of: mutex lock, spinlock, atomic operation and semaphore.

[0107] The synchronization mechanism refers to a mechanism used to coordinate access to shared resources by multiple parallel execution flows. Since the inference plane needs to read the weights and parallel policy flags of the current hybrid expert layer when performing inference computation, and the switching plane needs to update this shared data when performing policy switching steps, without protection, the inference plane may read partially updated weights or inconsistent flags, leading to inference computation errors. The synchronization mechanism can temporarily block the inference plane's access to this data when the switching plane updates shared data, and release it after the update is complete, ensuring that the inference plane always reads consistent data.

[0108] As an example, the synchronization mechanism can be implemented using a mutex lock. Before switching planes execute the activation of target weights and update of parallel policy tags, the mutex lock of the current hybrid expert layer is acquired; if the lock is already held by the inference plane, it waits; after acquiring the lock, the switching plane performs pointer swapping and tag update operations; the lock is released after the operations are completed. When the inference plane needs to read the weights or tags of the current hybrid expert layer, it also needs to acquire the same mutex lock, and release it after reading. Through the mutual exclusion access mechanism of the mutex lock, it is ensured that the inference plane cannot read data simultaneously when the switching plane updates data. As another example, the synchronization mechanism can be implemented using a spinlock, similar to a mutex lock, but without yielding processor time slices while waiting for the lock, suitable for scenarios with extremely short critical sections. As yet another example, the synchronization mechanism can be implemented using atomic operations, such as completing pointer swapping in a single instruction through hardware atomic swap instructions, without explicit locking. The synchronization mechanism can also be implemented using semaphores, controlling access permissions to shared resources through semaphores.

[0109] Among the synchronization mechanisms described above, mutexes are the most commonly used synchronization primitives, and their implementation is simple and reliable. Spin locks are suitable for scenarios with extremely short critical section operations and very short waiting times, avoiding the overhead of thread switching. Atomic operations are indivisible operations guaranteed at the hardware level, suitable for scenarios that update a single variable or pointer. Semaphores are suitable for controlling access to a limited number of shared resources by multiple execution flows. The specific synchronization mechanism chosen depends on the length of the critical section, concurrency requirements, and hardware support in the actual application scenario.

[0110] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application can ensure that the inference plane always reads consistent data by adopting a synchronization mechanism when updating weights and labels during plane switching. This avoids inconsistent weights or policy labels used in inference computation due to concurrent access conflicts and improves the reliability and stability of the system.

[0111] To further enable the current hybrid expert layer to instantly activate the new weights with extremely low overhead after loading, avoiding the additional GPU memory and time overhead caused by weight copying, in the hybrid expert sparse large model inference method provided in this application embodiment, step 110 of controlling the current hybrid expert layer to activate the target weights in the switching buffer specifically includes the following: The pointers to the original weights of the current hybrid expert layer are swapped with the pointers to the target weights in the switching buffer, and the parallel policy flag of the current hybrid expert layer is updated accordingly.

[0112] Correspondingly, see Figure 3 Step 110 may specifically include the following: Step 111: Load the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer; before the target weights are loaded, control the current hybrid expert layer to continue using its original weights to perform inference calculations; after the target weights are loaded, swap the pointers of the original weights of the current hybrid expert layer with the pointers of the target weights in the switching buffer, and update the parallel strategy flag of the current hybrid expert layer accordingly.

[0113] Step 112: Use the space occupied by the original weights as the switching buffer for the next hybrid expert layer.

[0114] As can be understood, the pointer swap refers to exchanging the pointer variable that points to the original weight's storage address with the pointer variable that points to the target weight's storage address in the switching buffer. In deep learning frameworks, model weights are typically stored as tensor objects, and the pointers point to the memory addresses of these tensor objects. After the swap operation, the pointer that previously pointed to the original weight now points to the target weight in the switching buffer, while the pointer that previously pointed to the target weight now points to the space occupied by the original weight. Since only pointers are swapped, rather than moving or copying the weight data itself, this operation has minimal time overhead and can be completed in a very short time, with almost no impact on inference computation.

[0115] As one example, the pointer swapping can be implemented using the tensor swapping interface provided by deep learning frameworks, such as swapping the data pointers of two tensors in the PyTorch framework. As another example, the pointer swapping can also be achieved by directly manipulating the underlying memory addresses, for example, by swapping the values ​​of two pointer variables.

[0116] Simultaneously with pointer swapping, the parallel policy flag of the current hybrid expert layer also needs to be updated. The parallel policy flag is a state variable that identifies the type of parallel policy adopted by the current hybrid expert layer. For example, when the target parallel policy is an expert parallel policy, the flag is updated to expert parallel; when the target parallel policy is a tensor parallel policy, the flag is updated to tensor parallel. During subsequent inference computation, the inference plane will select the corresponding parallel policy for inference based on this flag.

[0117] As an example, the parallel strategy flag can be an enumeration type variable, whose values ​​include expert parallelism and tensor parallelism. The update operation is to assign the variable's value to the corresponding enumeration value. As another example, the parallel strategy flag can also be a flag stored in shared memory. After the switching plane writes a new value, the inference plane obtains the latest flag value each time it reads.

[0118] The pointer swapping and tag update operations described above can be performed by the switching plane under the protection of the synchronization mechanism to avoid conflicts with the inference computation of the inference plane.

[0119] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application enables the activation of new weights through pointer swapping and tag updating. Only pointers need to be swapped without copying the weight data itself. The operation is completed instantaneously and has almost no impact on inference performance. At the same time, by updating the parallel policy tag, it is ensured that the inference plane can perform calculations according to the correct parallel policy in subsequent inference iterations.

[0120] To further reduce the memory usage of the accelerator processor during target weight loading and avoid pre-resident corresponding weight copies in the accelerator processor memory for each possible parallel strategy, in the hybrid expert sparse large model inference method provided in this application embodiment, the step 110 of loading the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer specifically includes the following: The target weights are loaded from the central processing unit memory into the switching buffer of the current hybrid expert layer; wherein the switching buffer is located in the video memory of the accelerated processor used to perform inference computation of the hybrid expert sparse large model.

[0121] Correspondingly, see Figure 4 Step 110 may specifically include the following: Step 113: Load the target weight from the central processing unit memory into the switching buffer of the current hybrid expert layer; before the target weight is loaded, control the current hybrid expert layer to continue using its original weight to perform inference computation; after the target weight is loaded, swap the pointer of the original weight of the current hybrid expert layer with the pointer of the target weight in the switching buffer, and update the parallel policy flag of the current hybrid expert layer accordingly; wherein, the switching buffer is located in the video memory of the accelerated processor used to perform inference computation of the hybrid expert sparse large model.

[0122] Step 112: Use the space occupied by the original weights as the switching buffer for the next hybrid expert layer.

[0123] It is understood that the central processing unit memory refers to the system main memory managed by the central processing unit (CPU), which typically has a large capacity and is used to store the complete weight data corresponding to all parallel strategies in a hybrid expert sparse large model. The accelerated processor refers to a dedicated processor used to perform inference computation, such as a graphics processing unit (GPU), whose video memory capacity is relatively limited but has a fast access speed, and is used to store weight data and intermediate computation results frequently used in the inference computation process.

[0124] During parallel policy switching, the weight data required by the target parallel policy is pre-stored in the CPU memory. When a policy switch is needed for the current hybrid expert layer, the target weights are loaded from the CPU memory into a switching buffer located in the accelerator processor's video memory. This loading method allows the accelerator processor's video memory to store only the weight data currently in use, without needing to maintain resident copies of the corresponding weights for multiple parallel policies simultaneously. After the switch is complete, the space occupied by the original weights is reused by the next hybrid expert layer, and the loaded target weights become the weights currently used by that layer, which can be overwritten or released when no longer needed.

[0125] As one example, the target weights can be transferred from CPU memory to accelerator memory via Direct Memory Access (DMA). The transfer process is controlled by the accelerator and requires no CPU intervention, resulting in high transfer efficiency. As another example, the target weights can also be transferred between CPU memory and accelerator memory using Unified Virtual Addressing (UVA), simplifying the programming model.

[0126] Compared to retrieving weight data from the memory of other accelerators via network or bus, loading target weights from CPU memory avoids consuming the memory resources and communication bandwidth of other accelerators, offering better scalability in large-scale distributed deployments. Compared to simultaneously maintaining weight copies in the accelerator's memory for multiple parallel strategies, the loading method in this embodiment significantly reduces memory overhead, making it particularly suitable for deployments with limited memory resources.

[0127] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application can avoid pre-resident corresponding weight copies in the accelerator processor memory by loading the target weights from the central processing unit memory to the switching buffer located in the accelerator processor memory, thereby reducing memory overhead; at the same time, the central processing unit memory has sufficient capacity to store the complete weight data corresponding to all parallel strategies, thereby improving the scalability of the system.

[0128] To further ensure that the parallel strategy switching in the decoding phase and the computation in the pre-filling phase do not interfere with each other, in the hybrid expert sparse large model inference method provided in this application embodiment, the hybrid expert sparse large model adopts an architecture that separates the pre-filling phase and the decoding phase. The computation in the pre-filling phase and the computation in the decoding phase are executed by different computing devices. See also... Figure 6 The computing device used to perform the calculations in the pre-filling stage can be called a pre-filling stage device, which transmits the key-value cache generated in the pre-filling stage to the computing device used to perform the calculations in the decoding stage. The computing device used to perform the calculations in the decoding stage can be called a decoding stage device, which includes an inference plane and a switching plane. The inference plane is used to perform inference calculations, and the switching plane is used to perform policy switching steps.

[0129] It is understood that, under the architecture of separating pre-filling and decoding, the method also includes the basic computational process of the decoding stage and the basic computational process of the pre-filling stage.

[0130] See Figure 4 Following step 200 in the decoding phase, the method further includes: when employing an expert parallel strategy, performing the following operations sequentially layer by layer on each hybrid expert layer of the hybrid expert sparse large model in each decoding iteration: Step 310: Obtain the attention output through multi-head attention calculation; Step 320: Obtain the expert ID for each word activation in the current batch using a gating function; Step 330: Based on the expert number, route each word to the computing device where the corresponding expert is located and perform expert calculation to generate the next word.

[0131] In the decoding phase, the current batch consists of the tokens generated by each in-process request in this decoding iteration. The multi-head attention calculation refers to the computational process of encoding input tokens using a multi-head self-attention mechanism, similar to traditional multi-head attention calculation methods. The gating function is a function used to calculate the affinity score of each token to each expert and select the top K experts with the highest affinity scores for activation. The routing refers to sending each token to the corresponding expert's computing device via a fully switched communication operation based on the expert number output by the gating function. The expert computation refers to the forward computation performed by each expert network on the tokens routed to its own device. In the decoding phase, each in-process request generates a new token in each decoding iteration, and all new tokens generated by in-process requests in this iteration constitute the current batch. For the specific implementation of the basic computational process under the expert parallel strategy in the decoding phase, please refer to the subsequent application examples; it will not be elaborated here.

[0132] See Figure 7 The pre-filling stage device is used to perform the following operations before the decoding stage of the hybrid expert sparse large model: Step 10: In the pre-filling stage of the hybrid expert sparse large model, receive the prompt word sequence input by the user; Step 20: Perform multi-head attention calculation layer by layer on each of the hybrid expert layers of the hybrid expert sparse large model; Step 30: Obtain the expert ID for each word activation in the current batch using a gating function; Step 40: Based on the expert number, route each lexical to the computing device where the corresponding expert is located and perform expert calculations to generate the first lexical and produce a key-value cache.

[0133] In this model, the hybrid expert layer of the hybrid expert sparse large model employs an expert parallel strategy in the pre-filling stage. In this pre-filling stage, the current batch is composed of the prompt word sequence. The prompt word sequence refers to the sequence of tokens obtained after word segmentation of the text input by the user into the hybrid expert sparse large model. The first token refers to the first output token generated after the pre-filling stage computation is completed. The key-value cache (KV-Cache) refers to the key and value vectors generated during the pre-filling stage computation, used in the autoregressive generation process of the decoding stage to avoid repeated computation of already processed inputs when generating new tokens. After the pre-filling stage computation is completed, the key-value cache is transmitted to the decoding stage computing device, and the decoding stage iteratively generates subsequent tokens based on the key-value cache. For a detailed implementation of the basic computation process under the expert parallel strategy in the pre-filling stage, please refer to the subsequent application examples.

[0134] As described above, the hybrid expert sparse large model inference method provided in this application adopts an architecture that separates the pre-filling stage and the decoding stage. The two stages are executed by different computing devices, and the parallel strategy switching in the decoding stage does not interfere with the computation in the pre-filling stage. Simultaneously, the computing device in the decoding stage has an inference plane and a switching plane, allowing inference computation and strategy switching to be asynchronous and parallel, further improving the stability and reliability of the inference service. Based on this architecture, both the pre-filling stage and the decoding stage follow the basic computational process of multi-head attention computation, gating functions, routing, and expert computation when employing expert parallel strategies, providing a foundation for subsequent load balancing optimization.

[0135] In expert parallelism for distributed inference of hybrid expert sparse large models, load imbalance caused by dynamic word routing is a key factor affecting inference performance. Dynamic word routing leads to uneven computational load, meaning different numbers of words are processed on each computing device. Simultaneously, the communication load of all-to-all communication operations is also uneven, meaning each computing device receives a different number of words. After the hybrid expert layer computation is complete, dynamic word routing during the reverse routing process also leads to uneven communication load, meaning different numbers of words are sent on each computing device. Therefore, load balancing scheduling of the words to be processed on each computing device is needed after the gating function obtains the expert number and before routing execution to alleviate the aforementioned load imbalance problem.

[0136] Based on this, in order to further address the problem of unbalanced computational load caused by dynamic word routing generated by the gating function in the expert parallel mode, and to avoid the lag and inaccuracy caused by relying on historical information for prediction load, a hybrid expert sparse large model inference method is provided in this application embodiment. Based on the above embodiment, see [link to embodiment]. Figure 4 The steps 320 and 330 executed by the decoding stage computing device may further include the following: Step 321: Based on the expert number activated by each word in the current batch, perform load balancing scheduling on each of the computing devices for the words to be processed.

[0137] The load balancing scheduling refers to adjusting the allocation of words based on the expert IDs activated by each word in the current batch before routing words to the computing devices where the corresponding experts reside, so that the number of words to be processed on each computing device tends to be balanced. Since the experts selected by the gating function for each word may be concentrated on some computing devices, causing some devices to be overloaded while others are idle, load balancing scheduling can alleviate this imbalance.

[0138] Accordingly, see Figure 7The steps 30 and 40 performed by the pre-filling stage computing device may further include the following: Step 31: Based on the expert number activated by each word in the current batch, perform load balancing scheduling on each of the computing devices for the words to be processed.

[0139] In the pre-filling phase, the computing device performs operations layer by layer on each hybrid expert layer. After obtaining the expert number activated by each word in the current batch through the gating function in step 30, and before routing each word to the computing device where the corresponding expert is located and performing expert calculations in step 40, the load balancing scheduling in step 31 is performed. In the pre-filling phase, the current batch consists of the input prompt word sequence.

[0140] During the decoding phase, when employing an expert parallel strategy, the computing device executes operations layer by layer on each hybrid expert layer. After obtaining the expert number activated by each word in the current batch through a gating function in step 320, and before routing each word to the computing device where the corresponding expert resides and performing expert computation in step 330, the load balancing scheduling in step 321 is performed. In the decoding phase, the current batch consists of words generated by each in-process request in this decoding iteration.

[0141] As one example, the load balancing scheduling can be achieved by counting the number of unprocessed tokens from local experts on each computing device and migrating excess tokens from computing devices exceeding the average token quota to computing devices with available quotas. As another example, the load balancing scheduling can also be achieved by adjusting the mapping relationship between tokens and experts, reallocating computing tasks without migrating token data. Specific implementation methods of load balancing scheduling will be described in detail in subsequent embodiments.

[0142] As described above, the hybrid expert sparse large model inference method provided in this application, by inserting a load balancing scheduling operation after obtaining the expert number in the gating function and before routing execution, can perform scheduling based on the real-time expert activation information of the current batch, avoiding the lag and insufficient accuracy problems caused by the prior art's reliance on historical information to predict the load. This load balancing scheduling operation is applicable when an expert parallel strategy is adopted in both the pre-filling stage and the decoding stage, and can maintain the balance of computational load in different inference stages.

[0143] Existing load balancing schemes typically predict expert load by recording historical information about word distribution and then adjust the expert load based on the prediction results. However, these load balancing optimization methods based on expert load prediction rely on periodically collected historical expert load information or offline statistics for load balancing adjustments, failing to provide real-time responses to expert load fluctuations, resulting in poor real-time performance and accuracy of load adjustments. In contrast, this embodiment acquires the expert activation information of the current batch in real time through a full collection operation, scheduling is entirely based on the real-time load information of the current batch, without requiring prediction based on historical data, and can respond to load fluctuations in real time.

[0144] Based on this, in order to further implement load balancing scheduling when scheduling is determined to be based on the output of the gating function, so as to minimize the number of missing experts after redistribution, and at the same time avoid occupying additional GPU memory space for load balancing, a hybrid expert sparse large model inference method provided in this application embodiment, based on the above embodiment, see [link to embodiment]. Figure 8 Step 321 and / or step 31 specifically includes the following: Step a: Obtain the expert IDs of all token activations in the current batch through a full collection operation.

[0145] The AllGather operation refers to a global communication operation among all computing devices participating in parallel expert computation. Each computing device broadcasts the expert IDs activated by each lexical unit in its current batch, stored locally, to all other computing devices, and simultaneously receives its own locally stored expert IDs from the other computing devices. Through the AllGather operation, each computing device can obtain complete information about the expert IDs activated by all lexical units in the current batch on each computing device. As an example, the AllGather operation can be implemented using the AllGather communication primitive.

[0146] Step b: Count the number of lexical units to be processed by local experts on each computing device, and sort the local experts according to the number of lexical units to obtain the sorting results.

[0147] After acquiring global expert activation information through the full collection operation, each computing device counts the total number of terms that each expert needs to process, i.e., the number of terms to be processed by each local expert. After the count is complete, each computing device sorts its local experts from lowest to highest based on the number of terms to be processed, resulting in a ranking. Experts ranked higher are considered low-load experts with fewer terms; these terms are retained for local computation. Experts ranked lower are considered high-load experts with more terms; any terms exceeding the average quota for these experts will be pruned and migrated.

[0148] Step c: Based on the average token quota of each computing device and the sorting result, the excess tokens on computing devices that exceed the average token quota are preferentially allocated to the computing device with the largest free quota in order to minimize the number of missing experts after load balancing.

[0149] The average token quota refers to the average number of tokens that each computing device should process, calculated by dividing the total number of tokens in the current batch by the total number of computing devices participating in the parallel computing of experts. When the number of tokens to be processed on a computing device exceeds the average token quota, the excess tokens are considered surplus tokens. Based on the sorting result obtained in step b, surplus tokens are preferentially pruned from the tokens corresponding to high-load experts. The pruned surplus tokens are preferentially allocated to the computing device with the largest free quota, which is the difference between the average token quota of the computing device and the number of tokens it currently retains. This greedy allocation strategy can allocate surplus tokens to computing devices with free space as much as possible, thereby minimizing the number of missing experts after load balancing.

[0150] The missing expert refers to an expert that needs to be processed on a computing device but is not stored locally on that device. Although the greedy allocation strategy can minimize the number of missing experts, since the distribution of experts across computing devices is fixed in advance, there may still be some terms that need to be processed by experts that are not stored locally after load balancing.

[0151] In the pre-filling stage, steps a to c above are completed by the pre-filling stage computing device after obtaining the expert number in step 30 and before performing routing in step 40. In the decoding stage, steps a to c above are completed by the decoding stage computing device after obtaining the expert number in step 320 and before performing routing in step 330.

[0152] The following example illustrates the load balancing scheduling process described above. (See [link]) Figure 9Assume there are 4 computing devices, each layer contains 8 experts, and 2 experts reside locally on each device. Device 0 hosts experts 0 and 1, Device 1 hosts experts 2 and 3, Device 2 hosts experts 4 and 5, and Device 3 hosts experts 6 and 7. The current batch contains 32 tokens, and each token activates 1 expert after being gated. After a full collection operation, the number of times each expert was activated is as follows: Expert 0 activated 9 times, Expert 1 activated 2 times, Expert 2 activated 6 times, Expert 3 activated 1 time, Expert 4 activated 2 times, Expert 5 activated 3 times, Expert 6 activated 0 times, and Expert 7 activated 9 times. The number of tokens to be processed on each device is calculated as follows: Device 0 has 11, Device 1 has 7, Device 2 has 5, and Device 3 has 9. The average token quota is 32 divided by 4, which equals 8. After sorting local experts by the number of tokens on each device from lowest to highest, device 0 retains 2 tokens from expert 1 locally, 6 tokens from expert 0 locally, and 3 tokens from expert 0 are pruned and migrated. Device 3 retains 8 tokens from expert 7 locally, and 1 token from expert 7 is pruned and migrated. The pruned excess tokens are preferentially allocated to device 2 with the largest available quota. Ultimately, the computational load of each device is 8 tokens, achieving complete balance.

[0153] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application obtains global expert activation information through a full collection operation, counts local expert load and sorts them by word count, and uses a greedy strategy based on the average word quota to preferentially allocate excess words to the computing device with the largest free quota. It can perform precise scheduling based on the real-time load information of the current batch, minimize the number of missing experts after load balancing, and provide a good foundation for subsequent on-demand loading of missing experts.

[0154] Existing load balancing solutions typically balance the load by deploying redundant expert copies on the computing device. However, this increases GPU memory usage during inference, which may offset performance gains in memory-constrained scenarios. In contrast, this embodiment employs an on-demand loading and immediate release mechanism. Missing experts are temporarily loaded from the CPU memory to the computing device's GPU memory and released immediately after the computation of the tokens involving the missing expert in the current batch is completed, rather than having redundant expert copies permanently resident in GPU memory. This significantly reduces GPU memory overhead.

[0155] Based on this, in order to further obtain the weights of missing experts for calculation even when there are still missing experts after load balancing scheduling, and to avoid redundant expert copies residing in video memory, in the hybrid expert sparse large model inference method provided in this application embodiment, on the basis of the above embodiment, after the load balancing scheduling of the tokens to be processed on each of the computing devices, and before the routing of each token to the computing device where the corresponding expert is located based on the expert number and before the expert calculation is performed, the method further includes the operation of loading and releasing missing experts on demand.

[0156] See Figure 8 Between steps 321 and 330 executed by the decoding stage computing device, or between steps 31 and 40 executed by the pre-filling stage computing device, after the load balancing scheduling is completed, the following content may also be included: Step d: If there are still missing experts after load balancing, the expert parameters of the missing experts are loaded from the central processing unit memory to the video memory of the corresponding computing device, and the loaded expert parameters are released after the word calculation involving the missing expert in the current batch is completed; wherein, the experts that need to be processed on the computing device and have been stored locally are the hit experts.

[0157] It is understood that the expert parameters refer to the model weight data corresponding to the missing expert, including parameters such as the weight matrix and bias vector of the expert network. The central processing unit memory refers to the system main memory managed by the central processing unit (CPU), which typically has a large capacity and is used to store the complete parameter data of all experts. The video memory of the computing device refers to the memory on the accelerated processor used to perform inference calculations; it has a fast access speed but relatively limited capacity.

[0158] After load balancing scheduling is completed, although the greedy allocation strategy can minimize the number of missing experts, since the distribution of experts across computing devices is fixed in advance, some terms may still need to be processed by experts who are not stored locally. For these missing experts, this embodiment temporarily loads their expert parameters from the central processing unit memory into the video memory of the corresponding computing device, instead of deploying redundant expert copies in the video memory in advance as in existing technologies. After all terms involving the missing expert in the current batch have been calculated, the loaded expert parameters are immediately released, and the occupied video memory space is returned, thereby avoiding the video memory overhead caused by redundant copies residing in video memory.

[0159] The "hit expert" refers to an expert that needs to be processed on a computing device and is already stored locally on that device. For hit experts, there is no need to load their expert parameters from the central processing unit's memory; the locally stored expert parameters can be used directly for calculation. By distinguishing between missing experts and hit experts, this embodiment can perform on-demand loading operations only for missing experts, while directly using local parameters for hit experts, reducing unnecessary data loading overhead.

[0160] As one example, the loading operation can transfer expert parameters from CPU memory to the video memory of the computing device via Direct Memory Access (DMA). As another example, the loading operation can also achieve data transfer between CPU memory and video memory via Unified Virtual Addressing (UVA). The releasing operation can return the video memory space occupied by the expert parameters to the video memory pool through the video memory management interface for use in subsequent operations.

[0161] In the pre-filling phase, the aforementioned on-demand loading and immediate release of missing experts is performed by the pre-filling phase computing device after completing the load balancing scheduling in step 31 and before executing the routing in step 40. In the decoding phase, the aforementioned operations are performed by the decoding phase computing device after completing the load balancing scheduling in step 321 and before executing the routing in step 330.

[0162] As described above, the hybrid expert sparse large model inference method provided in this application, when missing experts still exist after load balancing scheduling, loads the expert parameters of the missing experts from the central processing unit memory to the video memory of the computing device on demand, and releases them immediately after the computation of the terms involving the missing expert in the current batch is completed. This avoids the permanent presence of redundant expert copies in the video memory, reducing video memory overhead. Simultaneously, for hit experts, the locally stored parameters are used directly, reducing unnecessary data loading overhead. Compared with the prior art method of permanently maintaining redundant expert copies in video memory for load balancing, the on-demand loading and immediate release mechanism of this embodiment can significantly reduce video memory usage while ensuring computational correctness.

[0163] To further ensure data distribution consistency in subsequent multi-head attention calculations by restoring lexical units to the original computing device after expert computation in expert parallel mode, a hybrid expert sparse large model inference method provided in this application embodiment further includes a reverse routing operation after expert computation, based on the above embodiment.

[0164] See Figure 4 After step 330 executed by the decoding stage computing device, the following content may also be included: Step 340: Route each lexical back to the computing device where it was located before the gating function was executed via a full-switching communication operation for multi-head attention computation of the next hybrid expert layer.

[0165] See Figure 7 After step 40 performed by the pre-filling stage computing device, it may further include the following: Step 50: Route each lexical back to the computing device where it was located before the gating function was executed via a full-switching communication operation for multi-head attention computation of the next hybrid expert layer.

[0166] It is understood that the full-switching communication operation refers to a global communication operation between all computing devices participating in expert parallel computation. In expert parallel mode, after a word executes a gating function, it is routed to the computing device where the corresponding expert resides for expert computation. At this time, the word is distributed across different computing devices, disrupting the original data distribution. However, the multi-head attention layer adopts a data parallel approach, requiring each computing device to retain complete input data. Therefore, after the expert computation is completed, each word needs to be routed from the computing device currently executing the expert computation back to its original computing device before executing the gating function, restoring the original data distribution so that the multi-head attention computation of the next hybrid expert layer can be executed normally in a data parallel manner.

[0167] As an example, the fully switched communication operation can be implemented using the All-to-All communication primitive, where all computing devices simultaneously send tokens belonging to other computing devices to the target device and receive tokens belonging to their own computing devices from other devices. As another example, the fully switched communication operation can also first aggregate tokens to a specific computing device through a collection operation, and then distribute them to all original computing devices through a broadcast operation.

[0168] In the pre-filling stage, the aforementioned reverse routing operation is completed by the pre-filling stage computing device after performing the expert calculation in step 40. After routing the lexical back to the original computing device, the multi-head attention calculation in step 20 is performed on the next hybrid expert layer. In the decoding stage, the aforementioned reverse routing operation is completed by the decoding stage computing device after performing the expert calculation in step 330. After routing the lexical back to the original computing device, the multi-head attention calculation in step 310 is performed on the next hybrid expert layer.

[0169] As described above, the hybrid expert sparse large model inference method provided in this application restores the data distribution before the gating function was executed by performing a full-switch communication operation after the expert calculation is completed to route each term back to its original computing device. This ensures that the multi-head attention calculation of the next hybrid expert layer can be executed normally in a data-parallel manner. The reverse routing operation is a necessary step in the hybrid expert layer calculation process under the expert parallel mode, and together with the aforementioned load balancing scheduling and on-demand loading of missing experts, it constitutes a complete closed loop of expert parallel computing.

[0170] To further avoid loading overhead slowing down the overall inference performance when loading missing experts from the central processing unit memory on demand, in a hybrid expert sparse large model inference method provided in this application embodiment, based on the above embodiment, the process of loading the expert parameters of the missing expert from the central processing unit memory to the video memory of the corresponding computing device overlaps with the communication operation of routing each lexical to the computing device where the corresponding expert is located based on the expert number, and the calculation operation of the hit expert on the computing device.

[0171] It is understood that the overlapping execution refers to executing the loading process of missing experts, the communication operation of lexical routing, and the calculation operation of hit experts in parallel time, rather than sequentially. After load balancing scheduling is completed, each computing device may have both the lexical units that the hit experts need to calculate and the lexical units that the missing experts need to calculate. Since the hit experts have their expert parameters stored locally, they can start calculating directly; the missing experts need to load their expert parameters from the central processing unit's memory first, and the loading process requires a certain amount of time. If a sequential approach is used, loading the missing expert parameters first, then performing routing communication, and finally executing the hit expert calculation, the loading process will become a bottleneck in the pipeline, increasing the overall latency. This embodiment uses an overlapping execution approach, performing lexical routing communication and hit expert calculation operations simultaneously while loading the missing expert parameters, so that the time overhead of the loading process is masked by the communication and calculation processes, reducing the impact on the overall inference performance.

[0172] As an example, the overlapping execution can be implemented using an asynchronous loading mechanism. Specifically, an asynchronous loading task can be initiated in the accelerator processor to transfer the missing expert parameters from the CPU memory to the video memory. This asynchronous task executes in the background without blocking other computational and communication tasks on the accelerator processor. Simultaneously, the accelerator processor concurrently executes lexical routing communication operations and expert hit calculation operations. Once the asynchronous loading task is complete and the missing expert parameters are ready, the calculation operation for the missing expert can continue. As another example, the overlapping execution can also be implemented using multithreading or multiple execution streams, assigning the loading task, communication task, and computation task to different execution threads, with parallel scheduling performed by a hardware scheduler.

[0173] As one implementation, the overlapping execution can be achieved by submitting missing expert loading tasks to the accelerator's copy engine and computation tasks to the compute cores for parallel execution. The copy engine handles data transfer from CPU memory to GPU memory, while the compute cores perform expert computations; both can work concurrently on the same accelerator. While the copy engine executes loading tasks, the compute cores can perform communication operations and expert computations in parallel.

[0174] In the pre-filling phase, the aforementioned overlapping execution mechanism is applied by the pre-filling phase computing device in the routing and expert calculation operations of step 40 after performing load balancing scheduling from steps a to c. In the decoding phase, the aforementioned overlapping execution mechanism is applied by the decoding phase computing device in the routing and expert calculation operations of step 330 after performing load balancing scheduling from steps a to c.

[0175] As can be seen from the above description, the hybrid expert sparse large model inference method provided in this application overlaps the loading process of missing expert parameters with the word routing communication operation and the calculation operation of the hit expert, so that the time overhead of the loading process is masked by the communication and calculation process. This can avoid the loading operation becoming a bottleneck in the inference pipeline, reduce the impact on the overall inference performance, and improve inference efficiency.

[0176] In the pre-filling stage of hybrid expert sparse large models, a parallel expert strategy is typically employed. This involves calculating the affinity score of each term to each expert using a gating function, and then selecting the top K experts with the highest affinity scores for computation. However, dynamic term routing leads to unbalanced computational loads, meaning that different computing devices process different numbers of terms, and the communication load of the full-switching communication operation is also unbalanced. Existing load balancing schemes typically predict expert loads by recording historical term distribution information and then adjust the expert loads based on the prediction results. However, these load balancing optimization methods based on expert load prediction rely on periodically collected historical expert load information or offline statistics for load balancing adjustments, failing to provide real-time responses to expert load fluctuations, resulting in poor real-time performance and accuracy. Furthermore, existing schemes often balance the load by deploying redundant expert replicas on computing devices, which increases GPU memory usage during inference and may offset performance gains in memory-constrained scenarios.

[0177] To address at least one of the aforementioned problems, this application also provides a load balancing method for the pre-filling stage of a hybrid expert sparse large model. This method enables precise load balancing scheduling based on the real-time load information of the current batch during the pre-filling stage, without relying on historical predictions. Furthermore, by employing a mechanism for loading and releasing missing experts on demand, it avoids the permanent presence of redundant expert copies in video memory, thereby reducing video memory overhead.

[0178] Based on the pre-filling stage basic calculation process and load balancing optimization scheme described in the above embodiments, the load balancing method for the pre-filling stage of the hybrid expert sparse large model can be executed by the aforementioned pre-filling stage computing device, specifically including the following: Step F100: In the pre-filling stage of the hybrid expert sparse large model, the prompt word sequence input by the user is received, and the following operations are performed layer by layer on each hybrid expert layer of the hybrid expert sparse large model: multi-head attention calculation is performed, a gating function is executed, and the expert number of each word activated in the current batch is obtained; wherein, the current batch is composed of the input prompt word sequence, and the hybrid expert layer of the hybrid expert sparse large model adopts an expert parallel strategy in the pre-filling stage.

[0179] It is understood that the prompt word sequence refers to the sequence of lexical units obtained after the text input by the user to the hybrid expert sparse large model has been processed by word segmentation. The gating function is used to calculate the affinity score of each lexical unit to each expert, and select the top K experts with the highest affinity scores for activation. In the pre-filling stage, the hybrid expert layer adopts an expert parallel strategy by default, so the gating function is needed to determine which expert's computing device each lexical unit should be routed to.

[0180] Step F200: After obtaining the expert number activated by each word in the current batch through the gating function, and before routing each word to the corresponding computing device where the expert is located and performing expert calculation based on the expert number, load balancing scheduling is performed on the words to be processed on each computing device based on the expert number activated by each word in the current batch.

[0181] The load balancing scheduling refers to adjusting the allocation of words based on the expert IDs activated for each word in the current batch, before routing words to the computing devices where the corresponding experts reside. This ensures a more balanced number of words to be processed on each computing device. Since the gating function may select experts for each word on a few computing devices, leading to overload on some devices while others are idle, load balancing scheduling can alleviate this imbalance. As an example, load balancing scheduling can be achieved by counting the number of words to be processed by local experts on each computing device and migrating excess words from computing devices exceeding the average word quota to those with available quotas. Alternatively, load balancing scheduling can be achieved by adjusting the mapping relationship between words and experts, reallocating computing tasks without migrating word data.

[0182] Step F300: Based on the load balancing scheduling results, each lexical unit is routed to the computing device where the corresponding expert is located and expert calculations are performed to generate the first lexical unit and produce a key-value cache.

[0183] In this step, routing refers to sending each lexicon to the corresponding expert's computing device via full-switching communication operations based on the lexicon-expert mapping relationship after load balancing. Expert computation refers to each expert network performing forward computation on the lexicons routed to its device. After the pre-filling stage computation is completed, the first output lexicon is generated and a key-value cache is created. This key-value cache is used for the autoregressive generation process in the subsequent decoding stage.

[0184] To further illustrate a specific implementation of the above load balancing scheduling, in step F200, the load balancing scheduling of the words to be processed on each computing device based on the expert number activated by each word in the current batch may specifically include the following: Step F210: Obtain the expert IDs of all token activations in the current batch through a full collection operation.

[0185] The AllGather operation refers to a global communication operation among all computing devices participating in parallel expert computation. Each computing device broadcasts the expert IDs activated by each lexical term in the current batch, stored locally, to all other computing devices, while simultaneously receiving its own locally stored expert IDs from other computing devices. Through the AllGather operation, each computing device obtains complete information about the expert IDs activated by all lexical terms in the current batch across all computing devices. As an example, the AllGather operation can be implemented using the AllGather communication primitive. Unlike existing technologies that rely on historical information to predict load, the AllGather operation obtains real-time expert activation information for the current batch, accurately reflecting the load distribution of the current batch.

[0186] Step F220: Count the number of lexical units to be processed by local experts on each computing device, and sort the local experts according to the number of lexical units to obtain the sorting result.

[0187] After acquiring global expert activation information through the full collection operation, each computing device counts the total number of terms that each expert needs to process, i.e., the number of terms to be processed by each local expert. After the count is complete, each computing device sorts its local experts from lowest to highest based on the number of terms to be processed, resulting in a ranking. Experts ranked higher are considered low-load experts with fewer terms; these terms are retained for local computation. Experts ranked lower are considered high-load experts with more terms; any terms exceeding the average quota for these experts will be pruned and migrated.

[0188] Step F230: Based on the average token quota of each computing device and the sorting result, the excess tokens on computing devices that exceed the average token quota are preferentially allocated to the computing device with the largest free quota, so as to minimize the number of missing experts after load balancing.

[0189] The average token quota refers to the average number of tokens that each computing device should process, calculated by dividing the total number of tokens in the current batch by the total number of computing devices participating in parallel expert computation. When the number of tokens to be processed on a computing device exceeds the average token quota, the excess tokens are considered surplus tokens. Based on the sorting result obtained in step F220, surplus tokens are preferentially pruned from the tokens corresponding to high-load experts. The pruned surplus tokens are preferentially allocated to the computing device with the largest free quota, which is the difference between the average token quota of the computing device and the number of tokens it currently retains. This greedy allocation strategy can allocate surplus tokens to computing devices with free space as much as possible, thereby minimizing the number of missing experts after load balancing. The missing expert refers to an expert that needs to be processed on a computing device but is not stored locally on that computing device.

[0190] To further ensure that the weights of missing experts can still be obtained for calculation even after load balancing scheduling, and to avoid redundant expert copies residing in video memory, after step F230 above, and before routing each term to the computing device where the corresponding expert resides and performing expert calculation, the method may further include operations for loading and releasing missing experts on demand: Step F240: If there are still missing experts after load balancing, the expert parameters of the missing experts are loaded from the central processing unit memory to the video memory of the corresponding computing device, and the loaded expert parameters are released after the word calculation involving the missing expert in the current batch is completed; wherein, the experts that need to be processed on the computing device and have been stored locally are the hit experts.

[0191] The expert parameters refer to the model weight data corresponding to the missing expert, including parameters such as the weight matrix and bias vector of the expert network. After load balancing scheduling is completed, although the greedy allocation strategy can minimize the number of missing experts, since the distribution of experts on each computing device is fixed in advance, some terms may still need to be processed by experts that are not stored locally. For these missing experts, their expert parameters are temporarily loaded from the central processing unit memory into the video memory of the corresponding computing device, instead of deploying redundant expert copies in the video memory in advance. After all terms involving the missing expert in the current batch are calculated, the loaded expert parameters are immediately released, and the occupied video memory space is returned, thereby avoiding the video memory overhead caused by redundant copies residing in the video memory. Compared with the existing technology of residing redundant expert copies in the video memory to balance the load, this on-demand loading and instant release mechanism can significantly reduce video memory usage while ensuring computational correctness.

[0192] The "hit expert" refers to an expert that needs to be processed on a computing device and is already stored locally on that device. For hit experts, there is no need to load their expert parameters from the central processing unit's memory; the locally stored expert parameters can be used directly for calculation. By distinguishing between missing experts and hit experts, this embodiment can perform on-demand loading operations only for missing experts, while directly using local parameters for hit experts, reducing unnecessary data loading overhead.

[0193] As one example, the loading operation can transfer expert parameters from CPU memory to the video memory of the computing device via Direct Memory Access (DMA). As another example, the loading operation can also achieve data transfer between CPU memory and video memory via Unified Virtual Addressing (UVA). The releasing operation can return the video memory space occupied by the expert parameters to the video memory pool through the video memory management interface for use in subsequent operations.

[0194] To further mitigate the impact of loading overhead on overall inference performance when loading missing experts from CPU memory on demand, the process of loading the expert parameters of the missing experts from CPU memory to the video memory of the corresponding computing device can be overlapped with the communication operation that routes each term to the corresponding computing device based on the expert number, and the computation operation on the computing device for the hit expert. This overlapped execution means that the loading process is executed in parallel with the communication and computation operations, so that the time overhead of the loading process is masked by the communication and computation processes, reducing the impact on overall inference performance. Specifically, this can be achieved through an asynchronous loading mechanism or a multi-threaded approach, for example, submitting the loading task to the replication engine of the accelerator processor while the computing cores execute the communication and computation operations in parallel.

[0195] To restore the lexical units to the original computing device after expert computation in expert parallel mode, and to ensure data distribution consistency in subsequent multi-head attention computation, the method may further include a reverse routing operation after expert computation is performed in step F300. Step F400: Route each lexical back to the computing device where it was located before the gating function was executed via a full-switching communication operation for multi-head attention computation of the next hybrid expert layer.

[0196] Understandably, in expert parallel mode, after executing the gating function, tokens are routed to the computing devices of the corresponding experts for expert computation. At this point, tokens are distributed across different computing devices, disrupting the original data distribution. However, the multi-head attention layer employs data parallelism, requiring each computing device to store complete input data. Therefore, after expert computation is complete, each token needs to be routed back to its original computing device to restore the original data distribution. As an example, the fully switched communication operation can be implemented using the All-to-All communication primitive, where all computing devices simultaneously send tokens belonging to other computing devices to the target device and receive tokens belonging to their own computing device from other devices.

[0197] As described above, the load balancing method for the pre-filling stage of the hybrid expert sparse large model provided in this application avoids the problems of lag and insufficient accuracy caused by relying on historical information to predict the load by performing load balancing scheduling on the unprocessed terms on each computing device based on the real-time expert activation information of the current batch after obtaining the expert number in the gating function and before routing execution, thus achieving real-time response to load fluctuations. By obtaining global expert activation information through a full collection operation, statistically analyzing local expert load and sorting it by term quantity, and using a greedy strategy based on the average term quota to prioritize the allocation of surplus terms to the computing device with the largest available quota, the number of missing experts after load balancing can be minimized. For missing experts that still exist after load balancing, an on-demand loading and immediate release mechanism is adopted. The expert parameters of the missing experts are temporarily loaded from the central processing unit memory to the video memory of the computing device, and released immediately after the calculation of terms involving the missing expert in the current batch is completed. This avoids the permanent presence of redundant expert copies in the video memory and reduces video memory overhead. Meanwhile, by overlapping the loading process of missing expert parameters with the word routing communication operation and the calculation operation of the expert hit, the time overhead of the loading process is masked by the communication and calculation processes, which can prevent the loading operation from becoming a bottleneck in the inference pipeline and improve inference efficiency. In addition, by performing a full-switch communication operation after the expert calculation is completed to route each word back to its original computing device, the data distribution before the execution of the gating function can be restored, ensuring that the multi-head attention calculation of the next hybrid expert layer can be executed normally in a data-parallel manner.

[0198] In other words, the load balancing method for the pre-filling stage of a hybrid expert sparse large model provided in the above embodiments of this application relates to the field of computer technology. This load balancing method includes: in the pre-filling stage, executing a gating function layer by layer on each hybrid expert layer to obtain the expert number activated by each term in the current batch; after obtaining the expert number but before executing routing, performing load balancing scheduling on the terms to be processed on each computing device based on the real-time expert activation information of the current batch; and executing routing and expert computation based on the scheduling results. Specifically, the load balancing scheduling includes: obtaining global expert activation information through a full collection operation, statistically analyzing the local expert load and sorting it by the number of terms, and using a greedy strategy based on the average term quota to preferentially allocate excess terms to the computing device with the largest available quota. For missing experts that still exist after load balancing, they are loaded from the central processing unit memory on demand and released immediately after use to avoid resident redundant expert copies. The loading process overlaps with communication computation operations to hide loading overhead. This method can perform accurate scheduling based on real-time load information, reduce memory overhead, and improve inference performance.

[0199] From a software perspective, this application also provides a decoding stage computing device for performing the computational content of the decoding stage in the hybrid expert sparse large model inference method, see [link to relevant documentation]. Figure 5The decoding stage computing device includes an inference plane 01 and a switching plane 02. The decoding stage computing device includes a policy switching module 03, which is deployed in the switching plane 02. During the decoding stage of the hybrid expert sparse large model, in response to decision data indicating the switching of the current parallel policy to the target parallel policy, the policy switching step is sequentially executed on each hybrid expert layer of the hybrid expert sparse large model to switch each hybrid expert layer to the target parallel policy layer by layer.

[0200] It is understood that the policy switching module 03 is a functional module in the switching plane 02 responsible for executing the policy switching steps, and can be implemented by software, hardware, or firmware. The decision data refers to the instructions or signals used to trigger parallel policy switching. After acquiring the decision data, the policy switching module 03 initiates the layer-by-layer switching process. As an example, the policy switching module can be an independent execution thread, running in the switching plane 02, executing in parallel with the inference calculation thread in the inference plane 01.

[0201] It should be noted that when the policy switching module 03 performs the above switching steps, the inference plane continues to perform inference computation according to the current parallel policy tags of each hybrid expert layer. The policy switching module 03 is only responsible for updating the parallel policy tags, and the inference plane 01 selects the corresponding parallel policy according to the new tags in subsequent inference iterations. The two coordinate through the parallel policy tags; the policy switching module 03 updates the tags, and the inference plane 01 reads the tags. As an example, when the policy switching module 03 performs pointer swapping and tag updating, a synchronization mechanism can be used to protect it from conflicts with the inference computation of the inference plane 01.

[0202] As described above, the decoding stage computing device provided in this application embodiment has an inference plane and a switching plane. By deploying a policy switching module in the switching plane, policy switching steps are sequentially executed for each hybrid expert layer, enabling asynchronous layer-by-layer switching. The target weights required for the target parallel policy are loaded into the switching buffer of the current hybrid expert layer. Before loading is complete, the current hybrid expert layer continues to use its original weights to perform inference calculations. After loading is complete, new weights are enabled, and the space occupied by the original weights is used as the switching buffer for the next layer. Simultaneously, the remaining hybrid expert layers continue to perform inference calculations according to their respective parallel policies. Because the inference plane and the switching plane are isolated and asynchronously parallel, the policy switching operation of the switching plane will not block the inference service of the inference plane, ensuring uninterrupted inference service during parallel policy switching. Through the layer-by-layer cyclic reuse of the old weight space, it is possible to avoid having corresponding weight copies resident in the accelerated processor's memory for multiple parallel policies simultaneously, reducing memory overhead.

[0203] Based on the pre-filling stage basic calculation process and / or load balancing optimization scheme described in the above embodiments, this application also provides an embodiment of a pre-filling stage computing device for performing the calculation of the pre-filling stage in the hybrid expert sparse large model inference method and / or the load balancing method of the pre-filling stage of the hybrid expert sparse large model.

[0204] See Figure 6 The pre-filling stage computing device includes an inference computing module 04 and a transmission module 05.

[0205] The inference computation module 04 is used to receive the prompt word sequence input by the user during the pre-filling stage of the hybrid expert sparse large model, and to perform the following operations layer by layer on each hybrid expert layer of the hybrid expert sparse large model: perform multi-head attention computation, obtain the expert number activated by each word in the current batch through a gating function, route each word to the computing device where the corresponding expert is located based on the expert number and perform expert computation to generate the first word and generate a key-value cache.

[0206] The inference computation module 04 is the core functional module in the pre-filling stage computing device, used to execute the complete forward computation process of the pre-filling stage. The prompt word sequence refers to the sequence of tokens obtained after word segmentation of the text input by the user to the hybrid expert sparse large model. The hybrid expert layer of the hybrid expert sparse large model employs an expert parallel strategy in the pre-filling stage, and the current batch in the pre-filling stage is composed of the prompt word sequence. The inference computation module can be implemented by software, hardware, or firmware. As an example, the inference computation module can run on an accelerated processor, implementing multi-head attention computation, gating functions, and expert computation by calling the inference interface of a deep learning framework. As another example, the inference computation module can also be a dedicated hardware acceleration circuit, directly executing the above computational operations through hardware logic.

[0207] It is understood that the inference calculation module 04 can further perform load balancing scheduling operations when performing calculations layer by layer on each hybrid expert layer. Specifically, after obtaining the expert number activated by each word in the current batch through the gating function, and before routing each word to the corresponding expert's computing device based on the expert number and performing expert calculations, the inference calculation module 04 can also perform load balancing scheduling on the words to be processed on each computing device based on the expert number activated by each word in the current batch. The specific implementation of the load balancing scheduling has been described in detail in the above method embodiments and will not be repeated here. By performing load balancing scheduling, the problem of unbalanced computing load caused by the expert set selected by the gating function for each word can be alleviated.

[0208] After load balancing scheduling is completed, if missing experts still exist, the inference calculation module 04 can load the missing experts from the central processing unit memory to the video memory of the corresponding computing device, and release the loaded missing experts after the word calculation involving the missing expert in the current batch is completed. The loading process can be overlapped with the communication operation of routing each word to the computing device where the corresponding expert is located based on the expert number, and the calculation operation of the matching expert on the computing device, so as to hide the loading overhead.

[0209] After completing the expert calculation, the inference calculation module 04 can also route each lexical back to the computing device where it was located before the gating function was executed via a full-switching communication operation, in order to perform multi-head attention calculation for the next hybrid expert layer. The specific implementation of the reverse routing operation has been described in the above method embodiments.

[0210] The transmission module 05 is used to transmit the key-value cache generated in the pre-filling stage to the computing device in the decoding stage.

[0211] The transmission module 05 is a functional module in the pre-filling stage computing device responsible for data interaction with the decoding stage computing device. After the pre-filling stage calculation is completed, the transmission module transmits the key-value cache generated by the inference calculation module to the decoding stage computing device. The decoding stage then uses this key-value cache to iteratively generate subsequent tokens. As an example, the transmission module 05 can directly transmit the key-value cache from the video memory of the pre-filling stage computing device to the video memory of the decoding stage computing device via a high-speed network interface (such as InfiniBand, RoCE, etc.). As another example, the transmission module can also achieve data transmission through shared memory or an inter-node communication interface.

[0212] As described above, the pre-filling stage computing device provided in this application embodiment receives the prompt word sequence during the pre-filling stage through the inference computing module and performs multi-head attention calculation, gating function, and expert calculation layer by layer, generating the first lexical unit and producing a key-value cache. During the calculation process, the inference computing module can further perform load balancing scheduling based on real-time awareness and on-demand loading of missing experts, alleviating the load imbalance problem in the expert parallel mode and avoiding the permanent presence of redundant expert copies in the video memory, thus reducing video memory overhead. The key-value cache is transmitted to the decoding stage computing device through the transmission module, and the decoding stage can perform subsequent autoregressive generation based on the key-value cache, thereby realizing separate parallel inference between the pre-filling stage and the decoding stage.

[0213] Based on the decoding stage computing device, the pre-filling stage computing device, and their collaborative working method described in the above embodiments, this application also provides an embodiment of a hybrid expert sparse large model inference system. See also Figure 6The hybrid expert sparse large model inference system includes a decoding stage computing device, which contains an inference plane 01 and a switching plane 02.

[0214] The specific structure and function of the decoding stage computing device have been described in detail in the embodiments of the decoding stage computing device described above, and will not be repeated here. The inference plane 01 is used to perform inference computation of the hybrid expert sparse large model, that is, to perform multi-head attention computation and expert computation according to the current parallel strategy of each hybrid expert layer, and does not participate in parallel strategy switching. The switching plane 02 is used to sequentially perform strategy switching steps for each hybrid expert layer of the hybrid expert sparse large model, including loading the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer, controlling the current hybrid expert layer to continue to use its original weights to perform inference computation before the loading is completed, controlling the current hybrid expert layer to enable the new weights after the loading is completed and using the space occupied by the original weights as the switching buffer of the next hybrid expert layer, and controlling the remaining hybrid expert layers to continue to perform inference computation according to their respective parallel strategies.

[0215] The inference plane 01 and the switching plane 02 are isolated from each other and execute asynchronously in parallel. Switching operations on the switching plane 02 do not block the inference computation of the inference plane 01; the inference plane 01 only needs to execute the corresponding inference service according to the current parallel strategy of each hybrid expert layer. As an example, the inference plane 01 and the switching plane 02 can be two independent execution threads or processes, running on the same accelerator processor and interacting with each other through shared memory. As another example, the inference plane 01 and the switching plane 02 can run on different accelerator processors, communicating and coordinating through message passing or remote procedure calls.

[0216] In some embodiments, the hybrid expert sparse large model inference system further includes a pre-filling stage computing device, which is used to perform the pre-filling stage computing and / or the pre-filling stage load balancing method computing in the foregoing method embodiments.

[0217] The specific structure and function of the pre-filling stage computing device have been described in detail in the embodiments of the pre-filling stage computing device described above, and will not be repeated here. Before the decoding stage of the hybrid expert sparse large model, the pre-filling stage computing device performs pre-filling stage calculations, receives the prompt word sequence input by the user, and performs multi-head attention calculations, gating functions, and expert calculations layer by layer on each hybrid expert layer to generate the first lexical unit and produce a key-value cache. The pre-filling stage computing device is also used to transmit the generated key-value cache to the decoding stage computing device, which then performs iterative generation of subsequent lexical units based on this key-value cache.

[0218] In embodiments including a pre-filling stage computing device, the hybrid expert sparse large model inference system employs an architecture that separates the pre-filling stage from the decoding stage. The computations in the pre-filling stage and the decoding stage are performed by separate pre-filling and decoding stage computing devices, which are distinct physical computing devices. This separation architecture ensures that the computational loads of the pre-filling and decoding stages do not interfere with each other, allowing each stage to focus on its respective optimization objective: the pre-filling stage focuses on optimizing first-word latency, while the decoding stage focuses on optimizing output word latency.

[0219] As described above, the hybrid expert sparse large model inference system provided in this application, through the isolation of the inference plane and switching plane in the decoding stage computing device, can achieve uninterrupted inference service during parallel strategy switching; and through the layer-by-layer cyclic reuse of the old weight space, it can reduce memory overhead. In embodiments including a pre-filling stage computing device, through the architecture of separating the pre-filling stage from the decoding stage, the computation of the two stages does not interfere with each other, and the first word delay and output word delay can be optimized respectively. At the same time, when performing pre-filling computation, the pre-filling stage computing device can also perform real-time aware load balancing scheduling and on-demand loading of missing experts, alleviating the load imbalance problem in the expert parallel mode and further reducing memory overhead.

[0220] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the hybrid expert sparse large model inference method mentioned in the above embodiments or the load balancing method in the pre-filling stage of the hybrid expert sparse large model. The processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and the memory via wired or wireless means.

[0221] This application also provides another electronic device, which may include a memory, multiple processors, and a computer program stored in the memory and capable of running on the multiple processors. When the multiple processors execute the computer program, one processor performs operations on the inference plane, and another processor performs operations on the switching plane, jointly implementing the hybrid expert sparse large model inference method described in the first aspect above.

[0222] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0223] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the hybrid expert sparse large model inference method and / or the load balancing method in the pre-filling stage of the hybrid expert sparse large model in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the hybrid expert sparse large model inference method and / or the load balancing method in the pre-filling stage of the hybrid expert sparse large model in the above method embodiments.

[0224] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0225] The one or more modules are stored in the memory, and when executed by the processor, they execute the hybrid expert sparse large model inference method and / or the load balancing method of the hybrid expert sparse large model pre-filling stage in the implementation embodiment.

[0226] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0227] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0228] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0229] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the aforementioned hybrid expert sparse large model inference method and / or the load balancing method in the hybrid expert sparse large model pre-filling stage. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0230] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned hybrid expert sparse large model inference method and / or the load balancing method of the hybrid expert sparse large model pre-filling stage.

[0231] To further illustrate the above embodiments, this application also provides a specific application example of the complete execution flow of a load balancing method for the pre-filling phase using a pre-filling phase computing device and a parallel strategy asynchronous switching method for the decoding phase using a decoding phase computing device.

[0232] In expert parallelism for distributed inference of hybrid expert sparse large models, load imbalance caused by dynamic word routing is a key factor affecting inference performance. Dynamic word routing leads to uneven computational load, meaning different numbers of words are processed on each computing device. Simultaneously, the communication load of the full-switching communication operation is also uneven, meaning different numbers of words are received on each computing device. After the hybrid expert layer computation is completed, dynamic word routing during the reverse routing process also leads to uneven communication load, meaning different numbers of words are sent on each computing device.

[0233] To address the aforementioned load imbalance issue, existing load balancing schemes typically predict expert load by recording historical information about lexical distribution, and then adjust the expert load based on the prediction results. This involves distributing the load of hot experts across different computing devices through redundant experts, expert replication, and migration. However, these load balancing optimization methods based on expert load prediction rely on periodically collected historical expert load information or offline statistics for load balancing adjustments, failing to respond in real-time to expert load fluctuations, resulting in poor real-time performance and accuracy. Deploying redundant expert replicas on computing devices to balance the load increases memory usage during inference, potentially negating performance gains in memory-constrained scenarios. Furthermore, these schemes primarily focus on optimizing computational load balancing, neglecting communication and memory access loads; in communication-intensive or memory-intensive scenarios, computational load balancing does not equate to optimal overall performance. Additionally, these schemes are optimized for static parallel strategies and cannot adaptively switch parallel strategies based on load conditions. Some studies require pausing the inference service during expert load adjustment, resulting in high expert redistribution overhead. While some studies reduce the overhead of a single expert load adjustment by limiting the cost of each expert migration, load adjustment still exhibits a certain degree of lag.

[0234] To address at least one of the problems in the prior art, this application provides the following technical solution: For the pre-filling stage, a load balancing optimization method based on word count sorting and on-demand expert loading is proposed. This method uses a greedy strategy to evenly distribute words across multiple computing devices, and then loads missing experts in real time, achieving accurate load balancing without relying on historical predictions or maintaining redundant expert copies in GPU memory. For the decoding stage, a load-aware asynchronous switching method for parallel strategies is proposed. This method maintains or adaptively switches to expert parallelism to achieve high throughput under high load conditions, and uses the same load balancing optimization strategy as the pre-filling stage. Under low load conditions, it maintains or adaptively switches to tensor parallelism to achieve low latency, achieving load balancing while balancing latency requirements under low load conditions and throughput requirements under high load conditions. The parallel strategy switching in the decoding stage is implemented asynchronously, ensuring the inference service continues to operate normally during the switching process without causing service interruption, resulting in low switching overhead. Unlike existing load balancing optimization schemes based on load prediction, this application uses real-time awareness of expert load for accurate load allocation, offering significant advantages in terms of load balancing accuracy and real-time load adjustment.

[0235] This application example will, in conjunction with specific hardware configurations and load scenarios, fully demonstrate the execution flow of the above technical solution in the pre-filling and decoding stages, so as to facilitate those skilled in the art to understand the technical solution of this application. Specific details are as follows: (I) Basic Principles This application example proposes a dynamic parallel inference method and service system for hybrid expert sparse large model pre-filling-decoding (PD) separate parallel inference to solve the problem of unbalanced expert load. This application example proposes differentiated load balancing optimization strategies for the pre-filling and decoding stages, based on their respective load characteristics. Through core mechanisms such as real-time load awareness, on-demand expert loading, asynchronous switching of parallel strategies, and overlapping communication and computation, it systematically addresses several shortcomings in existing load balancing optimization techniques, thereby achieving better inference performance. The core technical principles are as follows: Existing technologies rely on historical expert load information or offline analysis for expert redistribution, which has significant time lag and large expert load prediction errors, potentially leading to performance degradation due to prediction bias. This application example proposes a load balancing optimization method based on word count sorting and on-demand expert loading for the pre-filling stage. After the gating function is executed, an AllGather operation is used to collect the expert IDs of all token activations in the current batch. The local expert load on each computing device is sorted from low to high according to the number of tokens to be processed. The average token quota for each computing device is calculated, and excess tokens on over-quota computing devices are greedily redistributed, prioritizing the allocation of expert loads with high excess quotas to computing devices with high available quotas, minimizing the number of missing experts. For missing experts not stored locally on the computing device, they are temporarily loaded from CPU memory to GPU memory for computation as needed, eliminating the need to pre-deploy redundant expert copies as in existing technologies, thus avoiding additional GPU memory usage. The loading of missing experts overlaps with the computation of locally shared experts, the computation of hit experts, and All-to-All communication for token routing, effectively hiding the loading overhead of missing experts. This method is entirely based on the real-time load information of the current batch, without requiring prediction based on historical data, and can respond to load fluctuations in real time, solving the lag problem of load balancing optimization in existing technologies. Furthermore, loading missing experts on demand, rather than resident redundant expert copies in GPU memory, significantly reduces GPU memory overhead.

[0236] Existing technologies optimize load balancing for static parallel strategies, consistently employing either Tensor Parallelism (TP) or Expert Parallelism (EP), which cannot adapt to the dynamic changes in batch size during the decoding phase. This application example proposes a load-aware asynchronous switching method for parallel strategies during the decoding phase. This method maintains or adaptively switches to EP under high load conditions to achieve high throughput, while employing the same load balancing optimization strategy as the pre-filling phase. Under low load conditions, it maintains or adaptively switches to TP to achieve low latency, achieving load balancing while balancing latency requirements under low load and throughput requirements under high load. In load awareness, an n-bit finite state machine is used to identify load. A high load is only identified when high loads are detected multiple times recently, and similarly, a low load is only identified when low loads are detected multiple times recently, thus avoiding frequent switching of parallel strategies due to instantaneous load oscillations. Parallel strategy switching during the decoding phase is implemented asynchronously. The inference service continues to operate normally during the switching process, without causing service interruption, resulting in low switching overhead. To achieve asynchronous parallel strategy switching, this application example divides the decoding phase into two isolated planes: the inference plane and the switching plane. The inference plane is responsible for normal inference service execution and does not participate in parallel strategy switching. The switching plane runs independently in the background and is responsible for load detection and identification, deciding whether to switch parallel strategies, and asynchronous switching of parallel strategies layer by layer.

[0237] The specific scheme for asynchronous switching of parallel strategies layer by layer is as follows: Before the strategy switching of the first hybrid expert layer, a switching buffer of appropriate size is first allocated on the GPU memory. Then, the weights required by the new parallel strategy are loaded from CPU memory into the switching buffer. Next, the pointers of the switching buffer and the old model weight tensor are swapped. That is, the switching buffer becomes the model weights required after the parallel strategy switching, while the space occupied by the old model weight tensor serves as the switching buffer for the next hybrid expert layer. The parallel strategy flag of the current hybrid expert layer is updated. The pointer swapping and parallel strategy flag update are protected by a mutex lock to avoid conflicts with the execution of the inference plane. The strategy switching process of each subsequent hybrid expert layer is similar to that of the first layer. After the strategy switching of the last hybrid expert layer is completed, the currently held switching buffer resources are released. The pointer swapping and parallel strategy flag update can be completed quickly, so the inference plane is almost unaware of the background parallel strategy switching. The inference plane only needs to execute the corresponding inference service according to the parallel strategy flag of the current hybrid expert layer. Therefore, the parallel strategy switching process has almost no impact on inference performance, and the overhead of parallel strategy switching is low.

[0238] This application example proposes a load balancing optimization method for hybrid expert sparse large model PD-based split parallel inference. The overall execution flow is as follows: Figure 10As shown, a PD-separated parallel inference system typically includes a pre-filling stage and a decoding stage, with different computing devices used for each stage. After the current batch of computation in the pre-filling stage is completed, the KV-Cache is transferred to the decoding stage computing device, which then generates the next term based on the current KV-Cache. This application example proposes targeted load balancing optimization methods for the two different inference stages of pre-filling and decoding. For the pre-filling stage and the high-load decoding stage using expert parallelism, this application example proposes a load balancing optimization method based on term quantity sorting and on-demand expert loading. This ensures that the number of terms processed by each expert on each computing device is the same, and that the terms processed by each computing device match the local experts as much as possible. For missing experts, they are temporarily loaded from CPU memory to GPU memory, and the temporarily loaded experts are released after term processing is completed. To address the low computational density of the decoding phase, this application example proposes a load-aware parallel strategy with asynchronous switching: Before each iteration in the decoding phase, the current batch size is sampled to detect and identify the load; under high load conditions, expert parallelism is maintained or adaptively switched to achieve high throughput, employing the same load balancing optimization strategy as the pre-filling phase; under low load conditions, tensor parallelism is maintained or adaptively switched to achieve low latency, thus achieving load balancing while simultaneously addressing latency requirements under low load conditions and throughput requirements under high load conditions. The specific scheme is as follows: (II) Load balancing optimization method based on word count ranking and expert on-demand loading For the pre-filling stage and high-load decoding stage employing expert parallelism, this application example proposes a load balancing optimization method based on word count ranking and on-demand expert loading. An example of its implementation is provided below. Figure 9 As shown. In expert parallelism, experts are evenly distributed across various computing devices. Figure 9 Eight experts are evenly distributed across four computing devices. After executing the gating function, the expert ID activated by each word is obtained. The first step involves collecting the expert IDs activated by all words in the current batch using an AllGather operation. Based on this, the number of words processed by the local experts on each computing device is calculated (e.g., ...). Figure 9 The first step involves two local experts on device 0: Expert 0 and Expert 1, who process 9 and 2 tokens respectively. The second step sorts the local expert load on each computing device from lowest to highest based on the number of tokens to be processed. The third step uses a greedy strategy to balance the load across computing devices: the average token quota for each computing device is calculated. When the number of tokens to be processed on a computing device exceeds the average quota, the tokens processed by the low-load experts are kept for local computation, while the tokens processed by the high-load experts are pruned and migrated (e.g., ...). Figure 9In device 0, the tokens processed by expert 0 are pruned and migrated, while the tokens processed by expert 1 remain local. During token migration, the workload of experts with high excess tokens is preferentially allocated to computing devices with high available quotas (e.g., Figure 9 The fourth step is to count the missing experts on each computing device after the load migration (e.g., the excess tokens of device 0 are preferentially migrated to device 2). The purpose of this is to minimize the number of missing experts after the load migration. Figure 9 In this method, after load migration, the missing expert for device 1 is expert 7, and the missing expert for device 2 is expert 0. For missing experts not stored locally on the computing devices, they are temporarily loaded from CPU memory to GPU memory for computation as needed. After word processing is complete, the temporarily loaded experts are released. The loading of missing experts overlaps with the computation of locally shared experts, the computation of matched experts, and the All-to-All communication for word routing, effectively hiding the loading overhead of missing experts. This method is entirely based on the real-time load information of the current batch and does not require prediction based on historical data. It can respond to load fluctuations in real time and solve the lag problem of existing load balancing optimization technologies. At the same time, this method loads missing experts on demand and releases the temporarily loaded missing experts after word processing is completed, rather than keeping redundant expert copies resident in GPU memory, thus significantly reducing GPU memory overhead.

[0239] In other words, in expert parallel mode, after the hybrid expert layer executes the gating function, it obtains the expert ID activated by each word. The first step involves collecting the expert IDs activated by all words in the current batch through an AllGather operation, and then calculating the number of words processed by local experts on each computing device. The second step involves sorting the local expert load on each computing device from low to high according to the number of words to be processed. The third step involves load balancing across computing devices based on a greedy strategy: calculating the average word quota for each computing device; when the number of words to be processed on a computing device exceeds the average quota, words processed by low-load experts are kept for local computation, while words processed by high-load experts are pruned and migrated. During word migration, the load of experts with excess words is preferentially allocated to computing devices with high available quotas to minimize the number of missing experts after load migration. The fourth step involves calculating the missing experts on each computing device after load migration. For missing experts not stored locally on the computing device, they are temporarily loaded from CPU memory to GPU memory for computation as needed, and the temporarily loaded experts are released after word processing is complete. The loading of missing experts overlaps with the computation of locally shared experts, the computation of matched experts, and the All-to-All communication for lexical routing, effectively hiding the overhead of loading missing experts. This method is entirely based on the real-time load information of the current batch, without requiring prediction based on historical data, and can respond to load fluctuations in real time. Furthermore, this method loads missing experts on demand and releases temporarily loaded missing experts after lexical processing is complete, rather than keeping redundant expert copies permanently resident in GPU memory, thus significantly reducing GPU memory overhead.

[0240] (III) Asynchronous Switching Method for Parallel Strategies Based on Load Awareness To address the low computational density of the decoding phase, this application example proposes a load-aware asynchronous switching method for parallel strategies. This method maintains or adaptively switches to expert parallelism under high load conditions to achieve high throughput. It employs the same load balancing optimization strategy as the pre-filling phase to maintain or adaptively switch to tensor parallelism under low load conditions to achieve low latency. This achieves load balancing while simultaneously addressing latency requirements under low load and throughput requirements under high load. The asynchronous switching of parallel strategies in this application example is implemented by dividing the decoding phase into two isolated planes (an inference plane and a switching plane).

[0241] The execution algorithm of the inference plane is as follows Figure 11 As shown, the inference plane is responsible for the normal execution of inference services and does not participate in parallel strategy switching. Figure 11 Lines 1-2 initialize the parallelism mode for each layer and the model as a whole. Here, we assume that it is initialized as tensor parallelism, but it can also be initialized as expert parallelism. Figure 11Lines 8-16 of the code perform the corresponding inference computation according to the parallel mode of each layer. The computation process of each layer is protected by a mutex lock to prevent the parallel strategy of the current layer from switching during the computation. Specifically, the input is L, which represents the total number of hybrid expert layers in the model. During the initialization phase, the parallel strategy flag modes[L] of each layer is initialized to tensor parallelism (TP), indicating that the system runs in TP mode by default, but it can also be initialized to expert parallelism (EP). The parallel strategy flag mode of the overall model is initialized to TP. A mutex lock mutexes[L] is allocated to each layer to protect the parallel strategy flag and weight pointer of each layer to prevent conflicts with the switching plane. The inference plane runs continuously in the main loop, performing inference computation layer by layer in each decoding iteration. Traversing each layer of the model (line 7), the mutex lock of the current layer is locked first (line 8) to prevent the parallel strategy of that layer from being modified by the switching plane during the inference process. If the current layer is marked as EP mode (lines 9-12), then the hybrid expert layer calculation is performed in EP mode, and the load balancing optimization method of this application example is adopted; if the current layer is marked as TP mode (lines 13-15), then the hybrid expert layer calculation is performed in TP mode. After the calculation is completed, the current layer is unlocked (line 16), allowing the switching plane to modify the parallel policy flag or weight pointer of this layer.

[0242] Parallel strategy switching plane execution algorithm as follows Figure 12 As shown, the parallel strategy switching plane runs independently in the background, responsible for load detection and identification, determining whether to switch parallel strategies, and asynchronous switching of parallel strategies layer by layer. In the detection and identification process, an n-bit finite state machine is used to identify the load. A load is only identified as high when high loads are detected multiple times recently, and similarly, a load is only identified as low when low loads are detected multiple times recently, thus avoiding frequent switching of parallel strategies caused by instantaneous load oscillations. Figure 12 The algorithm input consists of the number of bits n in a finite state machine and a load threshold θ. The specific values ​​of n and θ can be set based on empirical data. Line 1 initializes the state variable S. Lines 4-10 are for load detection and identification. The system obtains the batch size of the current decoding iteration in real time. When the batch size is greater than the load threshold θ, the system uses S=min(S+1,2) to... n-1 The system considers the current load to be high; conversely, when the batch size is less than the load threshold θ, the system considers the current load to be low using S=max(S-1,0). Lines 11-34 determine whether to trigger the parallel strategy switching and the asynchronous layer-by-layer switching mechanism for the parallel strategy. Lines 11-22 specifically address this: when the state variable S grows to its maximum value of 2... nWhen the current mode is not Expert Parallelism (EP) and the value is -1, the system determines that it is in a high-load state and triggers a parallel strategy switch. Then, it asynchronously switches the hybrid expert layers from tensor parallelism to expert parallelism layer by layer. The specific steps of this asynchronous switching are as follows: Before the parallel strategy switch of the first hybrid expert layer, a switching buffer of the appropriate size is allocated on the GPU memory (line 12). Then, the weights required for expert parallelism are loaded from CPU memory into this switching buffer (line 14). Next, the pointers of the switching buffer and the old model weight tensor are swapped (line 16). That is, the switching buffer becomes the model weights required for expert parallelism, and the space occupied by the old model weight tensor becomes the switching buffer for the next hybrid expert layer. The parallel strategy flag of the current hybrid expert layer is updated to EP (line 17). The pointer swap and parallel strategy flag update are protected by a mutex lock (lines 15 and 18) to avoid conflicts with the execution of the inference plane. The parallel strategy switching process of each subsequent hybrid expert layer is similar to that of the first layer. After the parallel strategy switch of the last hybrid expert layer is completed, the currently held switching buffer resources are released (line 20), and the overall parallel mode of the model is updated to EP (line 21). A schematic diagram illustrating the transition from tensor parallelism to layer-by-layer asynchronous switching to expert parallelism is shown below. Figure 13 As shown.

[0243] Figure 13 Taking four computing devices and four experts per layer as an example, this paper illustrates the process of switching from tensor parallelism (TP) to expert parallelism (EP) in each hybrid expert layer of the hybrid expert sparse large model from front to back while the inference service continues to run.

[0244] Figure 13 The diagram shows three layers in three states from top to bottom: layers that have switched to Expert Parallelism (EP) (such as layer 0 and layer i-1), layers that are executing policy switching steps (such as layer i), and layers that have not yet switched and are still maintaining Tensor Parallelism (TP) (such as layer i+1 and beyond). The switching order of each layer is to proceed layer by layer from the first layer.

[0245] according to Figure 12 The switching plane algorithm shown performs the following operations when the i-th layer switches from TP to EP: In the switching plane, the expert weights required by EP are loaded from CPU memory into the switching buffer of the layer, the layer is locked by a mutex lock, the pointers of the switching buffer and the old weights of the layer are swapped to enable the EP weights instantaneously, the parallel policy flag of the layer is updated to EP, and then the layer is unlocked.

[0246] Figure 12In lines 23-34, when the state variable S drops to its minimum value of 0 and the current mode is not Tensor Parallelism (TP), the system determines that it is in a low-load state and triggers a parallel policy switch. Then, it asynchronously switches the hybrid expert layers from expert parallelism to tensor parallelism layer by layer. The specific steps of this asynchronous switching are as follows: Before the parallel policy switch of the first hybrid expert layer, a switching buffer of the appropriate size is allocated on the GPU memory (line 24). Then, the weights required for tensor parallelism are loaded from CPU memory into this switching buffer (line 26). Next, the pointers of the switching buffer and the old model weight tensors are swapped (line 28). That is, the switching buffer becomes the model weights required for tensor parallelism, and the space occupied by the old model weight tensors becomes the switching buffer for the next hybrid expert layer. The parallel policy flag of the current hybrid expert layer is updated to TP (line 29). The pointer swap and parallel policy flag update are protected by mutex locks (lines 27 and 30) to avoid conflicts with the execution of the inference plane. The parallel policy switching process of each subsequent hybrid expert layer is similar to that of the first layer. After the parallel strategy switch of the last hybrid expert layer is completed, the currently held switching buffer resources are released (line 32), and the overall parallel mode of the model is updated to TP (line 33). A schematic diagram of the asynchronous layer-by-layer switch from expert parallelism to tensor parallelism is shown below. Figure 14 As shown.

[0247] Figure 14 and Figure 13 The switching direction is completely symmetrical. Taking four computing devices and four experts per layer as an example, it schematically shows the process of each hybrid expert layer of the hybrid expert sparse large model switching from expert parallelism (EP) to tensor parallelism (TP) from front to back while the inference service continues to run.

[0248] Figure 14 The diagram shows three layers in three states from top to bottom: layers that have switched to Tensor Parallelism (TP) (such as layer 0 and layer i-1), layers that are executing policy switching steps (such as layer i), and layers that have not yet switched and are still maintaining Expert Parallelism (EP) (such as layer L-1 and beyond). The switching order of each layer is to proceed layer by layer from the first layer.

[0249] according to Figure 12 The switching plane algorithm shown performs the following operations when the i-th layer switches from EP to TP: In the switching plane, the target weights required by TP are loaded from CPU memory into the switching buffer of the layer, the layer is locked by a mutex lock, the pointers of the switching buffer and the old weights of the layer are swapped to enable the TP weights instantaneously, the parallel policy mark of the layer is updated to TP, and then the layer is unlocked.

[0250] Where L represents the model layer number; i represents the layer index; n represents the number of bits; mode represents the overall parallel mode; modes[i] represents the parallel mode of the i-th layer; gbuffer_ep represents the expert parallel switching buffer; gbuffer_tp represents the tensor parallel switching buffer; lock(&mutexes[i]) represents locking the mutex of the i-th layer; unlock(&mutexes[i]) represents unlocking the mutex of the i-th layer.

[0251] In other words, in load detection and identification, an n-bit finite state machine is used to identify the load. A high load is only identified when high loads are detected multiple times recently, and similarly, a low load is only identified when low loads are detected multiple times recently. This avoids frequent switching of parallel strategies caused by instantaneous load oscillations. Under high load conditions, it maintains or adaptively switches to expert parallelism to achieve high throughput, and uses the aforementioned method based on word count sorting and expert on-demand loading for load balancing optimization. Under low load conditions, it maintains or adaptively switches to tensor parallelism to achieve low latency. This achieves load balancing while balancing latency requirements under low load conditions with throughput requirements under high load conditions. The asynchronous layer-by-layer switching of parallel strategies is achieved by dividing the decoding stage into two isolated planes (inference plane and switching plane). The switching process of the model parallel strategy is broken down into fine-grained local switching operations layer by layer. Each time, the parallel strategy is switched only for the current target layer. Each layer continues inference computation in the inference plane according to the current parallel mode, and inference requests are not blocked due to parallel strategy switching. During parallel policy switching at each layer, although pointer swapping and parallel policy tag updates are protected by mutex locks, these two operations can be completed quickly and have almost no impact on the execution performance of the inference plane. The inference plane only needs to execute the corresponding inference service according to the parallel policy tag of the current layer. The overhead of parallel policy switching layer by layer can be hidden by one or more decoding execution iterations. The inference service is not interrupted during parallel policy switching, and the inference plane is almost unaware of the asynchronous layer-by-layer parallel policy switching in the background. During parallel policy switching at each layer, only a switching buffer of appropriate size needs to be allocated on the GPU memory before the parallel policy switching of the first layer of hybrid expert layer. Then, the memory space is reused cyclically through pointer swapping. This reduces peak memory overhead and avoids frequent memory allocation and release operations on the GPU memory, further improving the efficiency of parallel policy switching.

[0252] The above-mentioned asynchronous switching mechanism for parallel strategies has the following advantages: (1) This mechanism breaks down the switching process of the model's parallel strategy into fine-grained local switching operations layer by layer. Each time, only the current target layer is switched. Each layer still performs inference calculations in the inference plane according to the current parallel mode, and the inference request will not be blocked due to the switching of parallel strategies. (2) Although the pointer swapping and parallel strategy mark update of each layer are protected by mutex locks, these two operations can be completed quickly and have almost no impact on the execution performance of the inference plane. The inference plane only needs to execute the corresponding inference service according to the parallel strategy mark of the current layer. The overhead of the layer-by-layer switching of parallel strategies can be hidden by one or more decoding execution iterations. The inference service will not be interrupted during the parallel strategy switching process, and the inference plane is almost unaware of the asynchronous switching of parallel strategies in the background. (3) Only a corresponding size of switching buffer needs to be allocated on the GPU memory before the parallel strategy switching of the first layer of the hybrid expert layer. Then, the memory space is reused in a circular manner through pointer swapping. This reduces the peak memory overhead and avoids frequent memory allocation and release operations on the memory, further improving the efficiency of parallel strategy switching.

[0253] Furthermore, the hybrid expert sparse large model inference system employs PD-separated parallel inference. During the pre-filling phase, the calculated KV-Cache is transmitted to the decoding phase computing device, which then performs subsequent inference calculations. For the pre-filling phase, the hybrid expert layer uses expert parallelism and the aforementioned load balancing optimization method based on lexical quantity sorting and on-demand expert loading. For the decoding phase, the aforementioned load-aware parallel strategy asynchronous switching method is used for optimization. Optimization in both phases does not interrupt the inference service system, achieving "uninterrupted service" online optimization. The on-demand expert loading in the pre-filling phase and the asynchronous switching of parallel strategies in the decoding phase are both hidden by the inference-related load.

[0254] In summary, compared with existing technologies, the load balancing optimization method based on dynamic parallelization provided in this application example has the following advantages: (1) Existing technologies such as Expert Parallelism Load Balancer (EPLB, proposed by the DeepSeek team in the "DeepSeek-V3 Technical Report" (arXiv:2412.19437, December 2024, and open-sourced in February 2025) and OmniPlacement (proposed by Huawei in the "OmniPlacement: Ascend Ultra-Large-Scale MoE Model Inference Load Balancing Technical Report" on May 20, 2025) are essentially "static or semi-static expert layout optimization" schemes. Their core idea is to balance the load on different computing devices as much as possible under the expert parallel framework through expert replication, expert migration, or redundant slots. Although these schemes can improve the load skew problem in the long-term statistical sense, their optimization goal is still limited to the computational balance "within the expert parallelism" and cannot dynamically change the parallel mode of the entire model according to the load of real-time services. This application example is different. It can not only achieve load balancing in expert parallel mode, but also dynamically switch between tensor parallel and expert parallel modes during the inference service operation based on real-time load information such as batch size. This allows the system to achieve lower latency in low-load scenarios and higher throughput in high-load scenarios.

[0255] (2) Regarding memory utilization, both EPLB and OmniPlacement employ redundant expert replication mechanisms to achieve load balancing. EPLB alleviates some computing device overload by replicating hotspot experts; OmniPlacement further introduces inter-layer redundant slots, enabling different layers to share redundant expert space. However, the essence of this type of method is still "trading extra memory for load balancing," so when the model size is large or GPU memory is limited, it will significantly increase memory pressure. Especially in hybrid expert models, expert parameters themselves occupy a large amount of storage space, and redundant replication will lead to repeated memory access and extra memory occupation, thereby weakening its benefits. This application example uses an expert on-demand loading and immediate release mechanism to replace the traditional redundant replication mechanism. Its core idea is not to keep redundant experts residing for a long time, but to dynamically determine whether experts need to be temporarily loaded based on the actual load of the current layer's terms. After the current layer's computation is completed, the memory resources of temporarily loaded experts will be released, thereby significantly reducing memory overhead.

[0256] (3) Both EPLB and OmniPlacement primarily focus on load balancing on the computation side, while neglecting communication and memory access issues. In actual inference processes, especially during the small-batch decoding stage, the system bottleneck is often no longer the amount of computation, but rather the communication and memory access latency. For example, when the batch size is small, even if expert computation is perfectly balanced, all-to-all communication across computing devices may still become the main bottleneck. In this case, continuing to use expert parallelism may lead to performance degradation due to excessive communication overhead. This application example proactively addresses this issue through a dynamic switching mechanism between tensor parallelism and expert parallelism: in low-batch-size scenarios during the decoding stage, the system automatically switches to tensor parallelism, allowing all computing devices to participate in the same matrix computation, thereby avoiding a large amount of all-to-all communication; while in high-batch-size scenarios during the decoding stage, due to the increase in batch size and computational density, the throughput advantage of expert parallelism begins to emerge, and the system switches back to expert parallelism. Therefore, this application example does not simply optimize the internal load of expert parallelism, but dynamically selects a more suitable parallel mode based on the current system load characteristics to improve the overall performance of the inference service system.

[0257] (4) Unlike EPLB and OmniPlacement, which require pausing inference services or reconstructing expert layouts, this application example further proposes a layer-by-layer asynchronous switching mechanism for parallel strategies, achieving "uninterrupted service" online parallel strategy switching. Traditional solutions often require pausing services or synchronously reconstructing expert mapping relationships when redeploying experts, resulting in a significant performance degradation during the switching process. In contrast, this application example breaks down the model parallel strategy switching process into layer-by-layer fine-grained local switching operations. Each time, only the current target layer is switched for parallel strategies, and each layer continues to perform inference computation in the inference plane according to the current parallel mode, without blocking inference requests due to parallel strategy switching. Although pointer swapping and parallel strategy mark updates at each layer are protected by mutex locks, these two operations can be completed quickly and have almost no impact on the execution performance of the inference plane. The inference plane only needs to execute the corresponding inference service according to the parallel strategy mark of the current layer. In addition, this application example only needs to allocate a switching buffer of the appropriate size on the GPU memory before switching the parallel strategy in the first layer of the hybrid expert layer. Then, the memory space is reused in a circular manner through pointer swapping. This reduces the peak memory overhead and avoids frequent memory allocation and release operations on the memory, further improving the efficiency of parallel strategy switching.

[0258] This application example has been implemented and its inference performance has been compared with EPLB and OmniPlacement on an 8-GPU A800 cluster with an inter-card bandwidth of 400GB / s. The experiment used the Qwen3.6-35B-A3B model. EPLB and OmniPlacement adopted an expert parallel strategy for the hybrid expert layer, and performed expert reshuffling every 300 inference iterations to optimize load balancing. The experimental results are shown in Figures 15(a) to 15(c). Figure 15(a) illustrates the comparison results of Output Term Latency (TPOT). TPOT measures the generation time of each output term in the decoding stage; the lower the value, the faster the model outputs. Figure 15(b) illustrates the comparison results of First Term Latency (TTFT). TTFT measures the time required from the user inputting a prompt word to the model generating the first output term; the lower the value, the lower the latency for the user to wait for the first response. Figure 15(c) illustrates the comparison results of system throughput. Throughput measures the total number of tokens a system can generate per second; a higher value indicates stronger system processing capabilities. Compared to EPLB and OmniPlacement, this application example reduces first-token latency (TTFT) by up to 1.17 times, output-token latency (TPOT) by up to 1.2 times, and increases inference system throughput by 1.11 times. This application example achieves significant performance gains in three key metrics—first-token latency, output-token latency, and inference throughput—compared to existing technologies.

[0259] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0260] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0261] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0262] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A hybrid expert sparse large model inference method, characterized in that, include: In the decoding phase of the hybrid expert sparse large model, in response to decision data indicating the switching of the current parallel strategy to the target parallel strategy, a strategy switching step is sequentially performed on each hybrid expert layer of the hybrid expert sparse large model to switch each hybrid expert layer to the target parallel strategy layer by layer. The strategy switching step includes: Load the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer; before the target weights are loaded, control the current hybrid expert layer to continue using its original weights to perform inference calculations; after the target weights are loaded, control the current hybrid expert layer to enable the target weights in the switching buffer, and use the space occupied by the original weights as the switching buffer of the next hybrid expert layer. In addition, control the remaining hybrid expert layers, other than the current hybrid expert layer, to continue performing inference computations according to their respective parallel strategies.

2. The hybrid expert sparse large model inference method according to claim 1, characterized in that, Before sequentially performing the policy switching step on each hybrid expert layer of the hybrid expert sparse large model, the method further includes: Continuously acquire the load index values ​​of the hybrid expert sparse large model during the decoding stage; If all the load index values ​​obtained consecutively are greater than the first load threshold or less than the second load threshold, then decision data is generated to indicate whether to switch the current parallel strategy to the target parallel strategy corresponding to the first load threshold or the target parallel strategy corresponding to the second load threshold; wherein, the first load threshold is greater than the second load threshold.

3. The hybrid expert sparse large model inference method according to claim 2, characterized in that, The target parallel strategy corresponding to the first load threshold is the expert parallel strategy, and the target parallel strategy corresponding to the second load threshold is the tensor parallel strategy. If all the load index values ​​obtained consecutively are greater than the first load threshold or less than the second load threshold, then decision data is generated to indicate whether to switch the current parallel strategy to the target parallel strategy corresponding to the first load threshold or the target parallel strategy corresponding to the second load threshold, including: A finite state machine is used, and the state values ​​of the finite state machine are updated according to a preset state update rule; wherein, the preset state update rule includes: when the load index value is greater than the first load threshold, the state value is increased; when the load index value is less than the second load threshold, the state value is decreased. If the state value reaches the first boundary value and the current parallel strategy is the tensor parallel strategy, then decision data is generated to indicate switching the current parallel strategy to the expert parallel strategy. If the state value reaches the second boundary value and the current parallel strategy is the expert parallel strategy, then decision data is generated to indicate switching the current parallel strategy to the tensor parallel strategy; wherein the first boundary value is greater than the second boundary value.

4. The hybrid expert sparse large model inference method according to claim 3, characterized in that, The second boundary value is 0; the first boundary value is 2. n -1, where n is an integer greater than 1; Increasing the state value includes: increasing the state value by 1; Decreasing the state value includes reducing the state value by 1.

5. The hybrid expert sparse large model inference method according to claim 2, characterized in that, The continuous acquisition of load metrics values ​​of the hybrid expert sparse large model during the decoding phase includes: In each decoding iteration of the decoding phase, the number of tokens generated by each in-process request in this decoding iteration is obtained, and the number of generated tokens is used as the load index value of this decoding iteration.

6. The hybrid expert sparse large model inference method according to claim 1, characterized in that, Before sequentially performing the policy switching step on each hybrid expert layer of the hybrid expert sparse large model, the method further includes: Allocate a corresponding switching buffer for the first hybrid expert layer in the hybrid expert sparse large model to execute the policy switching step; The method further includes: After the policy switching step is completed in the last hybrid expert layer of the hybrid expert sparse large model, the switching buffer corresponding to the last hybrid expert layer is released.

7. The hybrid expert sparse large model inference method according to claim 1, characterized in that, The decoding phase of the hybrid expert sparse large model is performed by the inference plane and the switching plane; The inference plane is used to perform inference computations for the hybrid expert sparse large model; The switching plane is used to sequentially execute policy switching steps on each hybrid expert layer of the hybrid expert sparse large model.

8. The hybrid expert sparse large model inference method according to claim 7, characterized in that, The switching plane employs a synchronization mechanism to execute the operation of enabling the target weight in the switching buffer for the current hybrid expert layer, and the operation of updating the parallel policy flag of the current hybrid expert layer, in order to avoid conflicts with the inference computation of the inference plane; The synchronization mechanism includes at least one of the following: mutex, spinlock, atomic operation, and semaphore.

9. The hybrid expert sparse large model inference method according to claim 1 or 8, characterized in that, The control of enabling the target weight in the switching buffer in the current hybrid expert layer includes: The pointers to the original weights of the current hybrid expert layer are swapped with the pointers to the target weights in the switching buffer, and the parallel policy flag of the current hybrid expert layer is updated accordingly.

10. The hybrid expert sparse large model inference method according to claim 1, characterized in that, The step of loading the target weights required by the target parallel strategy into the switching buffer of the current hybrid expert layer includes: The target weights are loaded from the central processing unit memory into the switching buffer of the current hybrid expert layer; wherein the switching buffer is located in the video memory of the accelerated processor used to perform inference computation of the hybrid expert sparse large model.

11. The hybrid expert sparse large model inference method according to claim 1, characterized in that, The hybrid expert sparse large model adopts an architecture that separates the pre-filling stage from the decoding stage; The calculations in the pre-filling stage and the calculations in the decoding stage are performed by different computing devices. The computing device used to perform the calculations in the pre-filling stage transmits the key-value cache generated in the pre-filling stage to the computing device used to perform the calculations in the decoding stage. The method further includes: In the decoding phase, when an expert parallel strategy is adopted, each hybrid expert layer of the hybrid expert sparse large model is executed layer by layer in each decoding iteration: attention output is obtained through multi-head attention calculation, expert number of each word activated in the current batch is obtained through gating function, and each word is routed to the computing device where the corresponding expert is located based on the expert number and expert calculation is performed to generate the next word; wherein, in the decoding phase, the current batch consists of words generated by each in-process request in this decoding iteration; The computing device used to perform the computation in the pre-filling stage is configured to perform the following before the decoding stage of the hybrid expert sparse large model: In the pre-filling stage of the hybrid expert sparse large model, receiving a sequence of prompt words input by the user, and sequentially performing multi-head attention computation on each hybrid expert layer of the hybrid expert sparse large model: obtaining the expert number activated by each lexical in the current batch through a gating function, and routing each lexical to the computing device where the corresponding expert is located based on the expert number and performing expert computation to generate the first lexical and produce a key-value cache; wherein, the hybrid expert layer of the hybrid expert sparse large model adopts an expert parallel strategy in the pre-filling stage, and in the pre-filling stage, the current batch is composed of the prompt word sequence.

12. The hybrid expert sparse large model inference method according to claim 11, characterized in that, After obtaining the expert IDs of each word activation in the current batch through the gating function, and before routing each word to the corresponding expert's computing device based on the expert IDs and performing expert calculations, the method further includes: Based on the expert IDs activated for each lexical unit in the current batch, load balancing scheduling is performed on the lexical units to be processed on each computing device.

13. The hybrid expert sparse large model inference method according to claim 12, characterized in that, The load balancing scheduling of unprocessed terms on each computing device based on the expert IDs activated for each term in the current batch includes: The expert IDs of all token activations in the current batch are obtained through a full collection operation; The number of lexical units to be processed by local experts on each of the aforementioned computing devices is counted, and the local experts are sorted according to the number of lexical units to obtain the sorting results; Based on the average token quota of each computing device and the sorting result, excess tokens on computing devices that exceed the average token quota are preferentially allocated to the computing device with the largest free quota in order to minimize the number of missing experts after load balancing; wherein, the missing expert is an expert that needs to be processed on a computing device but is not stored locally on that computing device.

14. The hybrid expert sparse large model inference method according to claim 13, characterized in that, After load balancing scheduling of the tokens to be processed on each of the computing devices, and before routing each token to the computing device where the corresponding expert is located based on the expert number and performing expert calculation, the method further includes: If a missing expert still exists after load balancing, the expert parameters of the missing expert are loaded from the CPU memory to the video memory of the corresponding computing device, and the loaded expert parameters are released after the word calculation involving the missing expert in the current batch is completed; wherein, the expert that needs to be processed on the computing device and has been stored locally is the hit expert.

15. The hybrid expert sparse large model inference method according to claim 14, characterized in that, The process of loading the expert parameters of the missing expert from the central processing unit memory to the video memory of the corresponding computing device overlaps with the communication operation of routing each word to the computing device where the corresponding expert is located based on the expert number, and the calculation operation on the computing device for the hit expert.

16. The hybrid expert sparse large model inference method according to claim 11, characterized in that, Following the expert calculations, the process also includes: Each lexical is routed back to the computing device where it was located before the gating function was executed via a fully switched communication operation for multi-head attention computation of the next hybrid expert layer.

17. A hybrid expert sparse large model inference system, characterized in that, A decoding stage computing device includes an inference plane and a switching plane for jointly executing the hybrid expert sparse large model inference method as described in claims 1 to 16; the inference plane is used to perform inference computation of the hybrid expert sparse large model; the switching plane is used to sequentially execute policy switching steps on each hybrid expert layer of the hybrid expert sparse large model.

18. An electronic device comprising a memory, a plurality of processors, and a computer program stored in the memory and executable on the plurality of processors, characterized in that, When multiple processors execute the computer program, one processor performs operations on the inference plane, and another processor performs operations on the switching plane, jointly implementing the hybrid expert sparse large model inference method as described in any one of claims 1 to 16.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the hybrid expert sparse large model inference method as described in any one of claims 1 to 16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the hybrid expert sparse large model inference method as described in any one of claims 1 to 16.

Images