Hybrid expert model load balancing adjustment method, device and storage medium
By acquiring predictive and historical expert fingerprints, the load of the hybrid expert model is dynamically adjusted, which solves the problem of load adjustment lag in the hybrid expert model, realizes forward-looking balanced load adjustment, and improves the timeliness and load balance of adjustment.
Patent Information
- Application Number
- CN202611305935.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-26
- Publication Date
- 2026-11-17
AI Technical Summary
Existing hybrid expert models exhibit lag in load adjustment, especially when traffic styles suddenly switch, resulting in poor timeliness of load adjustment and an inability to adjust load balancing in a timely manner.
By obtaining the predictive expert fingerprints of each waiting request in the waiting queue, obtaining the semantic load category based on the accurate expert fingerprints of historical running requests, and using clustered expert fingerprints to predict the expected expert load of the hybrid expert model, the deployment of experts on the graphics processor is dynamically adjusted to achieve forward-looking load balancing.
It improves the timeliness of load adjustment of the hybrid expert model, avoids load imbalance, and especially avoids the lag of traditional adjustment methods when traffic style changes suddenly, ensuring the early adjustment of expert layout and load balance.
Smart Images

Figure CN123114752A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of large language model reasoning and performance optimization, and in particular to a load balancing adjustment method, apparatus and storage medium for hybrid expert models. Background Technology
[0002] With the continuous development of Large Language Model (LLM) technology, Mixture-of-Experts (MoE) models have begun to come into view. How to solve the problem of unbalanced load in the inference stage of Mixture-of-Experts models has become an important issue in the field of model performance optimization.
[0003] In existing technologies, to address the problem of polarization between popular and unpopular experts during model inference, the graphics processing unit (GPU) where the expert is located is usually adjusted based on the number of times the expert has been called in history, according to the sliding window statistics, so as to keep the expert load on different GPUs balanced; or a load threshold is set, and when the expert load on a certain GPU exceeds the load threshold, the graphics processing unit where the expert is located is adjusted.
[0004] However, the adjustment method based on the number of historical expert calls requires the system to accumulate new data for a period of time before adjusting when the traffic style suddenly changes, resulting in a significant lag. Similarly, the adjustment method based on load thresholds can only achieve load adjustment when the graphics processor is overloaded, resulting in poor timeliness of load adjustment. Summary of the Invention
[0005] This invention provides a load balancing adjustment method, device, electronic device, storage medium, and computer program product based on a hybrid expert model to solve the problem of lag in the load balancing adjustment process.
[0006] According to another aspect of the present invention, a load balancing adjustment method based on a hybrid expert model is provided, comprising: Obtain the predicted expert fingerprint of each waiting request in the waiting queue; wherein the predicted expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model; The semantic load category is obtained based on the precise expert fingerprint of each historical running request, and the semantic load category corresponding to each waiting request is obtained based on the predicted expert fingerprint of each waiting request; wherein, the precise expert fingerprint reflects the expert distribution result of the historical running requests in the hybrid expert model; Based on the clustering expert fingerprints of the semantic load categories corresponding to each waiting request, the expected expert load of the hybrid expert model is obtained, so as to adjust the load of the hybrid expert model according to the expected expert load.
[0007] The step of obtaining the predictive expert fingerprint of each waiting request in the waiting queue includes: obtaining the predictive expert fingerprint of the current waiting request based on the expert distribution probability of a specified number of words in the current waiting request; wherein, the specified number of words is related to the number of words and word association relationships in the current waiting request.
[0008] The step of adjusting the load of the hybrid expert model according to the expected expert load includes: obtaining an overload threshold and an idle load threshold based on the global average load of each graphics processor and a preset adjustment coefficient; if it is determined that the expected load of the first graphics processor is greater than the overload threshold, creating a copy of the first expert with the lowest global expected load in the first graphics processor; if it is determined that the expected load of the second graphics processor is less than the idle load threshold, deleting a copy of the second expert with the lowest local expected load in the second graphics processor.
[0009] After adjusting the load of the hybrid expert model according to the expected expert load, the method further includes: obtaining the reduction amount of the difference between the maximum expected load and the minimum expected load of each graphics processor when each of the waiting requests is used as a new running request; obtaining the running score of each waiting request according to the queue addition time, running priority and the reduction amount of the difference, so as to select at least one target waiting request as the actual new running request based on the running score.
[0010] The step of selecting at least one target waiting request as an actual new running request based on the running score includes: sequentially selecting each target waiting request as an actual new running request until a preset constraint rule is met; wherein, the preset constraint rule includes at least one of the following: the total amount of resources occupied by the current running request reaches a preset resource threshold; there are no waiting requests in the waiting queue with a running score exceeding the preset score threshold; the load difference of each graphics processor exceeds a preset balancing threshold; there are no waiting requests in the waiting queue with a waiting time exceeding a preset time threshold.
[0011] After adjusting the load of the hybrid expert model according to the expected expert load, the method further includes: obtaining the expert distribution probability of the current running request in the deep layer of the hybrid expert model based on the expert distribution result of the current running request in the shallow layer of the hybrid expert model, and performing word segmentation processing based on the expert distribution probability of the deep layer of the hybrid expert model.
[0012] According to another aspect of the present invention, a load balancing adjustment device based on a hybrid expert model is provided, comprising: The predictive expert fingerprint acquisition module is used to acquire the predictive expert fingerprint of each waiting request in the waiting queue; wherein, the predictive expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model; The semantic load category acquisition module is used to acquire the semantic load category based on the accurate expert fingerprint of each historical running request, and to acquire the semantic load category corresponding to each waiting request based on the predicted expert fingerprint of each waiting request; wherein, the accurate expert fingerprint reflects the expert distribution result of the historical running requests in the hybrid expert model; The load balancing adjustment execution module is used to obtain the expected expert load of the hybrid expert model based on the clustering expert fingerprint of the semantic load category corresponding to each waiting request, so as to adjust the load of the hybrid expert model according to the expected expert load.
[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the load balancing adjustment method of the hybrid expert model as described in any embodiment of the present invention.
[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the load balancing adjustment method of the hybrid expert model as described in any embodiment of the present invention.
[0015] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the load balancing adjustment method of the hybrid expert model as described in any embodiment of the present invention.
[0016] The technical solution of this invention involves obtaining the predicted expert fingerprints of each waiting request in the waiting queue; obtaining the semantic load category based on the accurate expert fingerprints of each historical running request; obtaining the semantic load category corresponding to each waiting request based on the predicted expert fingerprints of each waiting request; and obtaining the expected expert load of the hybrid expert model based on the clustering expert fingerprints of the semantic load categories corresponding to each waiting request, so as to adjust the load of the hybrid expert model according to the expected expert load. This achieves forward-looking load balancing adjustment, ensuring the early adjustment of the expert layout of the hybrid expert model, improving the timeliness of the load adjustment process, avoiding load imbalance, and especially avoiding the adjustment lag present in traditional load balancing methods when traffic style changes abruptly.
[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart of a load balancing adjustment method using a hybrid expert model according to Embodiment 1 of the present invention; Figure 2 This is a flowchart of another load balancing adjustment method using a hybrid expert model provided in Embodiment 2 of the present invention; Figure 3 This is a flowchart of another load balancing adjustment method using a hybrid expert model provided in Embodiment 3 of the present invention; Figure 4 This is a schematic diagram of the structure of a load balancing adjustment device based on a hybrid expert model according to Embodiment 4 of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the load balancing adjustment method of the hybrid expert model in the embodiments of the present invention. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0022] Example 1 Figure 1 This is a flowchart of a load balancing adjustment method for a hybrid expert model provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where the load balancing of a hybrid expert model is pre-adjusted based on the predicted expert fingerprints of waiting requests. This method can be executed by a load balancing adjustment device for a hybrid expert model, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S101. Obtain the predicted expert fingerprint of each waiting request in the waiting queue; wherein, the predicted expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model.
[0023] The hybrid expert model decomposes the feedforward network into multiple smaller networks (i.e., experts). For each request input to the hybrid expert model, only a few experts participate in the calculation of each request's token, while other experts remain dormant and do not consume computing power. Therefore, under the premise of a large total number of parameters, the actual computational amount of a single inference is controllable. In the embodiments of the present invention, the hybrid expert model includes L (greater than or equal to 2) hybrid expert layers, and each hybrid expert layer includes E (greater than or equal to 2) experts. Tokens with different semantics prefer to activate different experts.
[0024] The predicted expert fingerprints for waiting requests can be obtained through a fingerprint prediction network. This network can consist of a two-layer MLP (Multilayer Perceptron) and a layer-wise attention pooling module. Since the MLP has only two layers, the dimensionality of the hidden layers is much smaller than the Transformer dimension of the hybrid expert model, resulting in very few parameters, simple computation, and minimal GPU resource consumption during inference, thus not hindering the execution of the main inference task. The layer-wise attention pooling module provides each hybrid expert layer with a separate set of attention query vectors. The waiting queue is a mechanism in the hybrid expert model used to store requests awaiting processing; requests enter the waiting queue to be activated and processed.
[0025] When a request enters the fingerprint prediction network, the network can process a specified number of tokens (e.g., 32), specifically the top 32 tokens. Each token undergoes two MLP layers to obtain a transformed token feature sequence. Then, through a layer attention pooling module, the token sequence is weighted and aggregated using L sets of independent attention parameters to produce L feature vectors. These L feature vectors are matched one-to-one with L hybrid expert layers. Next, the output projection layer predicts the expert distribution probabilities of each hybrid expert layer in E dimensions. Finally, the expert distribution probabilities of all hybrid expert layers are concatenated to output the complete predicted expert fingerprint.
[0026] For example, suppose a hybrid expert model has two hybrid expert layers (i.e., L equals 2), each hybrid expert layer has four experts (i.e., E equals 4), and the specified quantifier is 10. The hybrid expert layer L1 of the fingerprint prediction network predicts that 6 tokens will be assigned to expert 1, 3 tokens to expert 2, 0 tokens to expert 3, and 4 tokens to expert 4. Then the sub-fingerprint of hybrid expert layer L1 is [0.6, 0.3, 0.0, 0.1]. Similarly, suppose the sub-fingerprint of hybrid expert layer L2 is [0.7, 0.0, 0.1, 0.2]. Then the concatenation of sub-fingerprint 1 and sub-fingerprint 2 is the predicted expert fingerprint. Obviously, the predicted expert fingerprint reflects the expert preference distribution (i.e., expert distribution probability) of the current request in the hybrid expert model.
[0027] Specifically, after each waiting request is activated and processed, the expert distribution results (i.e., accurate expert fingerprints) after the actual execution of each waiting request can be recorded. All tokens (or a specified number of tokens) of that waiting request, along with the accurate expert fingerprints, are then used to form a training sample. This training sample is used to asynchronously fine-tune the fingerprint prediction network, thereby continuously improving the prediction accuracy of the fingerprint prediction network. Therefore, compared to obtaining the predicted expert fingerprint based on all tokens of the waiting request, obtaining the predicted expert fingerprint based on a specified number of words ensures both higher prediction accuracy and improved prediction efficiency.
[0028] Optionally, in this embodiment of the invention, obtaining the predictive expert fingerprint of each waiting request in the waiting queue includes: obtaining the predictive expert fingerprint of the current waiting request based on the expert distribution probability of a specified number of words in the current waiting request; wherein, the specified number of words is related to the number of words and word association relationships in the current waiting request.
[0029] Specifically, the more lexical units there are, the more lexical units are needed to fully reflect the semantics of the waiting request, and the specified number of lexical units needs to be configured to a larger value. Conversely, the fewer lexical units there are, the fewer lexical units are needed to fully reflect the semantics of the waiting request, and the specified number of lexical units can be configured to a smaller value. Lexical relationships reflect the semantic similarity between lexical units. When the similarity is high, a smaller number of lexical units can reflect the true semantics of the waiting request, and the specified number of lexical units can be configured to a smaller value. When the similarity is low, a larger number of lexical units are needed to reflect the true semantics of the waiting request, and the specified number of lexical units can be configured to a larger value. Therefore, by dynamically calculating the specified number of lexical units, both the accuracy of the predicted expert fingerprint of the waiting request is ensured, and the efficiency of obtaining the predicted expert fingerprint is improved.
[0030] S102. Obtain the semantic load category based on the precise expert fingerprint of each historical running request, and obtain the semantic load category corresponding to each waiting request based on the predicted expert fingerprint of each waiting request; wherein, the precise expert fingerprint reflects the expert distribution result of the historical running request in the hybrid expert model.
[0031] Running requests refer to requests that have been retrieved from the waiting queue and activated and processed by the hybrid expert model, i.e., requests that have entered the Prefill phase of the hybrid expert model; while historical running requests refer to requests that have undergone the Prefill and Decode phases of the hybrid expert model, i.e., requests that have been completed and responded to by the hybrid expert model. The precise expert fingerprints of each historical running request are obtained, and the precise expert fingerprints of each historical running request are clustered in real time to obtain different semantic load categories based on the expert type actually invoked by each token of each historical running request; for example, category 1 is code, category 2 is mathematical derivation, and category 3 is general chat.
[0032] Meanwhile, the average of the accurate expert fingerprints under each semantic load category is used as the clustering expert fingerprint of that semantic load category. Obviously, the clustering expert fingerprint represents the typical expert distribution under the current semantic load category. Then, the predicted expert fingerprint of each waiting request in the waiting queue is compared with the clustering expert fingerprint of each semantic load category to determine the semantic load category to which each waiting request belongs.
[0033] S103. Obtain the expected expert load of the hybrid expert model based on the clustering expert fingerprint of the semantic load category corresponding to each of the waiting requests, so as to adjust the load of the hybrid expert model according to the expected expert load.
[0034] Since requests of the same category have highly similar expert activation patterns, the clustered expert fingerprint of the semantic load category to which the waiting request belongs is used instead of the predicted expert fingerprint of the waiting request for global load accumulation. The accumulation result reflects the future load of each expert in the hybrid expert model (i.e., the predicted expert load), thereby reducing the amount of accumulation computation in large-scale concurrent scenarios and reducing the computational overhead of the scheduling process.
[0035] The expected expert load of the hybrid expert model is actually represented by a future load heatmap, which shows the distribution of traffic pressure for each expert over a period of time. Since an expert can be deployed on multiple GPUs simultaneously in the form of replicas to distribute the computing pressure, for experts with high expected load, the number of GPUs deployed can be increased to improve the token processing capacity of the expert. For experts with low expected load, the number of GPUs deployed can be reduced to reduce the resource consumption of the expert.
[0036] Optionally, in this embodiment of the invention, adjusting the load of the hybrid expert model according to the expected expert load includes: obtaining an overload threshold and an idle load threshold based on the global average load of each graphics processor and a preset adjustment coefficient; if it is determined that the expected load of the first graphics processor is greater than the overload threshold, creating a copy of the first expert with the lowest global expected load in the first graphics processor; if it is determined that the expected load of the second graphics processor is less than the idle load threshold, deleting a copy of the second expert with the lowest local expected load in the second graphics processor.
[0037] Specifically, for a single GPU, multiple copies of experts can be deployed on it. The expected load of each GPU can be determined by summing the expected loads of all experts deployed on a single GPU. The global average load refers to the average of the expected loads of all GPUs. The preset adjustment coefficient can be configured as needed (e.g., 0.2). The product of the preset adjustment coefficient and the global average load is used as the target adjustment threshold. The sum of the global average load and the target adjustment threshold is used as the overload threshold. The difference between the global average load and the target adjustment threshold is used as the idle load threshold.
[0038] When the expected load of a graphics processor (i.e., the first graphics processor) exceeds the overload threshold, it indicates that the first graphics processor may be overloaded. At this time, the coldest expert in the global prediction (i.e., the first expert), that is, the expert with the lowest expected global load, is selected, and a copy of the first expert is created in the first graphics processor. This allows the first graphics processor to simultaneously handle both popular and unpopular experts. Since the total global request traffic of unpopular experts is relatively small, even if a copy is added, the amount of computation diverted to this copy has a natural upper limit, which will not bring a huge increase in load. Therefore, by taking advantage of the low upper limit of traffic of unpopular experts, idle video memory is filled without creating new hot spots, thus smoothing out the global load difference.
[0039] When the expected load of a graphics processor (i.e., the second graphics processor) is less than the idle load threshold, it indicates that the second graphics processor may be idle. At this time, the copy of the second expert with the lowest expected load (i.e., the lowest local expected load) within the second graphics processor is selected and deleted, so that the second graphics processor only carries popular experts. This ensures that the video memory resources are not occupied by idle unpopular expert copies, avoids resource waste, and provides video memory resources to the popular experts that are already being carried.
[0040] The technical solution of this invention involves obtaining the predicted expert fingerprints of each waiting request in the waiting queue; obtaining the semantic load category based on the accurate expert fingerprints of each historical running request; obtaining the semantic load category corresponding to each waiting request based on the predicted expert fingerprints of each waiting request; and obtaining the expected expert load of the hybrid expert model based on the clustering expert fingerprints of the semantic load categories corresponding to each waiting request, so as to adjust the load of the hybrid expert model according to the expected expert load. This achieves forward-looking load balancing adjustment, ensuring the early adjustment of the expert layout of the hybrid expert model, improving the timeliness of the load adjustment process, avoiding load imbalance, and especially avoiding the adjustment lag present in traditional load balancing methods when traffic style changes abruptly.
[0041] Example 2 Figure 2 This is a flowchart of a load balancing adjustment method using a hybrid expert model provided in Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is that one or more requests are selected for a pre-filling stage while waiting for requests, such as... Figure 2 As shown, the method specifically includes: S201. Obtain the predicted expert fingerprint of each waiting request in the waiting queue; wherein, the predicted expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model.
[0042] S202. Obtain the semantic load category based on the precise expert fingerprint of each historical running request, and obtain the semantic load category corresponding to each waiting request based on the predicted expert fingerprint of each waiting request; wherein, the precise expert fingerprint reflects the expert distribution result of the historical running request in the hybrid expert model.
[0043] S203. Obtain the expected expert load of the hybrid expert model based on the clustering expert fingerprint of the semantic load category corresponding to each of the waiting requests, so as to adjust the load of the hybrid expert model according to the expected expert load.
[0044] S204. Obtain the amount by which the difference between the maximum expected load and the minimum expected load of each graphics processor is reduced when each of the waiting requests is used as a new running request.
[0045] Each waiting request in the waiting queue is assumed to be a new running request and added to the continuous batch processing framework. The impact of each waiting request on the expert load balance is then obtained. This is achieved by obtaining the expected load of each graphics processor after the request is added, and the difference between the maximum value (maximum expected load) and the minimum value (minimum expected load) is taken as the load difference. Obviously, the smaller the load difference, the better the load balance. Then, the load difference before and after the addition is compared to obtain the difference in load difference (i.e., the reduction). If the reduction in load difference is positive, it means that the load balance of the hybrid expert model is improved after the waiting request is added. If the reduction in load difference is negative, it means that the load balance of the hybrid expert model is further deteriorated after the waiting request is added.
[0046] S205. Based on the queue addition time, running priority, and difference reduction amount of each waiting request, obtain the running score of each waiting request, and select at least one target waiting request as the actual new running request based on the running score.
[0047] The longer the queue time for a currently waiting request is, the longer the waiting time for the current request is, and the current waiting request will be given priority as a new running request; the higher the running priority of the current waiting request, the greater the importance of the current waiting request, and the current waiting request will be given priority as a new running request; the greater the decrease in the difference of the current waiting request, the greater the improvement in the load balancing of the hybrid expert model after the current waiting request is added, and the current waiting request will also be given priority as a new running request.
[0048] Accordingly, the queue addition time, running priority, and the difference reduction amount of the current waiting request are multiplied by their respective weights. The results of each multiplication are then summed, and the sum is the running score of each waiting request. Obviously, the higher the running score, the higher the priority of the current waiting request as an actual new running request. Based on this, one or more waiting requests with higher running scores can be used as actual new running requests to complete the response to the waiting requests. This not only achieves the orderly response to waiting requests, but also ensures that waiting requests with earlier queue addition time, higher running priority, and the most significant improvement in load balancing are given priority processing.
[0049] Optionally, in this embodiment of the invention, selecting at least one target waiting request as an actual new running request based on the running score includes: sequentially selecting each target waiting request as an actual new running request until a preset constraint rule is met; wherein the preset constraint rule includes at least one of the following: the total amount of resources occupied by the current running request reaches a preset resource threshold; there are no waiting requests in the waiting queue with a running score exceeding the preset score threshold; the load difference of each graphics processor exceeds a preset balancing threshold; and there are no waiting requests in the waiting queue with a waiting time exceeding a preset time threshold.
[0050] Specifically, if the total GPU memory resources consumed by all running requests reach a preset resource threshold, the addition of new running requests is stopped; if all waiting requests with high running times in the waiting queue have been added to the new running request list, the addition of new running requests is stopped; if the load imbalance among the GPUs, i.e., the difference between the maximum expected load and the minimum expected load, exceeds a preset balancing threshold, the addition of new running requests is stopped; if all waiting requests with long waiting times in the waiting queue have already appeared as new running requests, i.e., there are no waiting requests with waiting times exceeding a preset time threshold, the addition of new running requests is stopped. This achieves batch processing of waiting requests, improves request processing efficiency, avoids excessive processing pressure on the hybrid expert model, and effectively balances the load of different GPUs in the hybrid expert model.
[0051] The technical solution of this invention, after adjusting the load of the hybrid expert model according to the expected expert load, obtains the reduction in the difference between the maximum and minimum expected load of each graphics processor when each waiting request is treated as a new running request; based on the queue addition time, running priority, and difference reduction of each waiting request, a running score is obtained for each waiting request, and at least one target waiting request is selected as the actual new running request based on the running score. This not only achieves orderly response to waiting requests, but also ensures priority processing of waiting requests with earlier queue addition time, higher running priority, and the most significant improvement in load balancing.
[0052] Example 3 Figure 3 This is a flowchart of a load balancing adjustment method for a hybrid expert model provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is that, based on the expert distribution results of the running request in the shallow layer of the hybrid expert model, the probability of its expert distribution in the deep layer of the hybrid expert model is predicted, such as... Figure 3 As shown, the method specifically includes: S301. Obtain the predicted expert fingerprint of each waiting request in the waiting queue; wherein, the predicted expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model.
[0053] S302. Obtain the semantic load category based on the precise expert fingerprint of each historical running request, and obtain the semantic load category corresponding to each waiting request based on the predicted expert fingerprint of each waiting request; wherein, the precise expert fingerprint reflects the expert distribution result of the historical running request in the hybrid expert model.
[0054] S303. Obtain the expected expert load of the hybrid expert model based on the clustering expert fingerprint of the semantic load category corresponding to each of the waiting requests, so as to adjust the load of the hybrid expert model according to the expected expert load.
[0055] S304. Based on the expert distribution results of the current running request in the shallow layer of the hybrid expert layer, obtain the expert distribution probability of the current running request in the deep layer of the hybrid expert layer, and perform word segmentation processing based on the expert distribution probability of the deep layer of the hybrid expert layer.
[0056] Because the hybrid expert model uses a top-down serial computation mechanism, each word first passes through the shallow layer (i.e., the shallow layer of the hybrid model, such as layers 1, 2, and 3) and then enters the deep layer (i.e., the deep layer of the hybrid model). Therefore, after the hybrid expert model completes the shallow layer computation, the expert distribution results of the shallow layer can be obtained. At this time, the deep layer has not yet performed routing computation. The expert distribution results of the shallow layer can be sent to a lightweight Transformer network. This Transformer network can be composed of two layers and has the characteristics of small hidden dimension and low computational cost. In this way, the correlation between the shallow routing pattern and the deep expert activation can be learned. Based on this, the Transformer network can be used to predict how many words each expert in each deep layer is expected to receive.
[0057] When the number of tokens received by an expert on a GPU exceeds its computing power limit, a token overflow operation is triggered. The overflowed tokens are migrated to other replicas of that expert to share the token processing burden of the previous expert. During the migration, priority is given to migrating to expert replicas on other GPUs within the same node (e.g., a server) to reduce communication latency. If the current node does not have a replica of the expert, then the expert replica is migrated across nodes to GPUs on other nodes. In this way, the overflow decision is based on a cross-layer lookahead mechanism to achieve global view prediction.
[0058] Specifically, token migration consumes GPU interconnect bandwidth, causing communication latency. In this case, the token migration threshold can be flexibly configured. If the overall load of the current batch of running requests is relatively balanced, a higher token migration threshold should be configured, meaning tokens are only migrated when the expert load is severely excessive, reducing frequent data transfers and saving communication overhead. If the overall load of the current batch of running requests is unbalanced, with severe load skew and prominent hot experts, a lower token migration threshold should be configured, ensuring high migration efficiency for overflowing tokens and improving the timeliness of load balancing.
[0059] The technical solution of this invention, after adjusting the load of the hybrid expert model according to the expected expert load, obtains the expert distribution probability of the current running request in the deep layer of the hybrid expert model based on the expert distribution results of the current running request in the shallow layer of the hybrid expert model, and performs word segmentation processing based on the expert distribution probability of the deep layer of the hybrid expert model. Thus, the word segmentation processing in the deep layer of the hybrid expert model, based on the expert distribution results of the shallow layer of the hybrid expert model, further avoids the generation of hotspot experts and effectively balances the load pressure of different graphics processors.
[0060] Example 4 Figure 4 This is a structural block diagram of a load balancing adjustment device based on a hybrid expert model provided in Embodiment 4 of the present invention. The device specifically includes: The predictive expert fingerprint acquisition module 401 is used to acquire the predictive expert fingerprint of each waiting request in the waiting queue; wherein, the predictive expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model; The semantic load category acquisition module 402 is used to acquire the semantic load category based on the accurate expert fingerprint of each historical running request, and to acquire the semantic load category corresponding to each waiting request based on the predicted expert fingerprint of each waiting request; wherein, the accurate expert fingerprint reflects the expert distribution result of the historical running request in the hybrid expert model; The load balancing adjustment execution module 403 is used to obtain the expected expert load of the hybrid expert model based on the clustering expert fingerprint of the semantic load category corresponding to each waiting request, so as to adjust the load of the hybrid expert model according to the expected expert load.
[0061] The technical solution of this invention involves obtaining the predicted expert fingerprints of each waiting request in the waiting queue; obtaining the semantic load category based on the accurate expert fingerprints of each historical running request; obtaining the semantic load category corresponding to each waiting request based on the predicted expert fingerprints of each waiting request; and obtaining the expected expert load of the hybrid expert model based on the clustering expert fingerprints of the semantic load categories corresponding to each waiting request, so as to adjust the load of the hybrid expert model according to the expected expert load. This achieves forward-looking load balancing adjustment, ensuring the early adjustment of the expert layout of the hybrid expert model, improving the timeliness of the load adjustment process, avoiding load imbalance, and especially avoiding the adjustment lag present in traditional load balancing methods when traffic style changes abruptly.
[0062] Optionally, the predictive expert fingerprint acquisition module 401 is specifically used to acquire the predictive expert fingerprint of the current waiting request based on the expert distribution probability of a specified number of words in the current waiting request; wherein, the specified number of words is related to the number of words and word association relationships in the current waiting request.
[0063] Optionally, the load balancing adjustment execution module 403 is specifically used to obtain an overload threshold and an idle load threshold based on the global average load of each graphics processor and a preset adjustment coefficient; if it is determined that the expected load of the first graphics processor is greater than the overload threshold, a copy of the first expert with the lowest global expected load is created in the first graphics processor; if it is determined that the expected load of the second graphics processor is less than the idle load threshold, a copy of the second expert with the lowest local expected load in the second graphics processor is deleted.
[0064] Optionally, the load balancing adjustment device of the hybrid expert model is further configured to obtain the reduction amount of the difference between the maximum expected load and the minimum expected load of each graphics processor when each of the waiting requests is respectively used as a new running request; and to obtain the running score of each of the waiting requests based on the queue addition time, running priority and the reduction amount of the difference, so as to select at least one target waiting request as the actual new running request based on the running score.
[0065] Optionally, the load balancing adjustment device of the hybrid expert model is specifically used to sequentially treat each target waiting request as an actual new running request until a preset constraint rule is met; wherein, the preset constraint rule includes at least one of the following: the total amount of resources occupied by the current running request reaches a preset resource threshold; there are no waiting requests in the waiting queue with a running score exceeding a preset score threshold; the load difference of each graphics processor exceeds a preset balancing threshold; there are no waiting requests in the waiting queue with a waiting time exceeding a preset time threshold.
[0066] Optionally, the load balancing adjustment device of the hybrid expert model is further configured to obtain the expert distribution probability of the current running request in the deep layer of the hybrid expert model based on the expert distribution result of the current running request in the shallow layer of the hybrid expert model, and perform word segmentation processing based on the expert distribution probability of the deep layer of the hybrid expert model.
[0067] The above-described apparatus can execute the load balancing adjustment method of the hybrid expert model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the load balancing adjustment method of the hybrid expert model provided in any embodiment of the present invention.
[0068] Example 5 Figure 5 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, electronic devices, blade electronic devices, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0069] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0070] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0071] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as load balancing conditioning methods using hybrid expert models.
[0072] In some embodiments, the hybrid expert model load balancing tuning method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed on a heterogeneous hardware accelerator via ROM and / or a communication unit. When the computer program is loaded into RAM and executed by a processor, one or more steps of the hybrid expert model load balancing tuning method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform the hybrid expert model load balancing tuning method by any other suitable means (e.g., by means of firmware).
[0073] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0074] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0075] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0076] To provide user interaction, the systems and techniques described herein can be implemented on a heterogeneous hardware accelerator, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the heterogeneous hardware accelerator. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and input from the user can be received in any form (including sound input, voice input, or haptic input).
[0077] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0078] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0079] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0080] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A load balancing adjustment method using a hybrid expert model, characterized in that, include: Obtain the predicted expert fingerprint of each waiting request in the waiting queue; wherein the predicted expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model; The semantic load category is obtained based on the precise expert fingerprint of each historical running request, and the semantic load category corresponding to each waiting request is obtained based on the predicted expert fingerprint of each waiting request; wherein, the precise expert fingerprint reflects the expert distribution result of the historical running requests in the hybrid expert model; Based on the clustering expert fingerprints of the semantic load categories corresponding to each waiting request, the expected expert load of the hybrid expert model is obtained, so as to adjust the load of the hybrid expert model according to the expected expert load.
2. The load balancing adjustment method based on a hybrid expert model according to claim 1, characterized in that, The step of obtaining the prediction expert fingerprint of each waiting request in the waiting queue includes: Based on the expert distribution probability of a specified number of terms in the current waiting request, obtain the predicted expert fingerprint of the current waiting request; wherein, the specified number of terms is related to the number of terms and the term association relationship in the current waiting request.
3. The load balancing adjustment method based on a hybrid expert model according to claim 1, characterized in that, Adjusting the load of the hybrid expert model according to the expected expert load includes: Based on the global average load of each graphics processor and the preset adjustment coefficient, obtain the overload load threshold and idle load threshold. If it is determined that the expected load of the first graphics processor is greater than the overload threshold, a copy of the first expert with the lowest global expected load is created in the first graphics processor. If it is determined that the expected load of the second graphics processor is less than the idle load threshold, the copy of the second expert with the lowest local expected load in the second graphics processor is deleted.
4. The load balancing adjustment method based on a hybrid expert model according to claim 1, characterized in that, After adjusting the load of the hybrid expert model according to the expected expert load, the method further includes: When each of the waiting requests is used as a new running request, the difference between the maximum expected load and the minimum expected load of each graphics processor is reduced. Based on the queue addition time, running priority, and difference reduction amount of each waiting request, a running score is obtained for each waiting request, and at least one target waiting request is selected as an actual new running request based on the running score.
5. The load balancing adjustment method based on a hybrid expert model according to claim 4, characterized in that, The step of selecting at least one target waiting request as an actual new running request based on the running score includes: Each target waiting request is sequentially treated as a new running request until a preset constraint rule is met; wherein the preset constraint rule includes at least one of the following: The total resources currently being used by running requests have reached the preset resource threshold; There are no waiting requests in the waiting queue whose running scores exceed the preset score threshold; The load difference among the various graphics processors exceeds the preset balancing threshold; There are no waiting requests in the waiting queue whose waiting time exceeds a preset time threshold.
6. The load balancing adjustment method based on a hybrid expert model according to claim 1, characterized in that, After adjusting the load of the hybrid expert model according to the expected expert load, the method further includes: Based on the expert distribution results of the current running request in the shallow layer of the hybrid expert layer, obtain the expert distribution probability of the current running request in the deep layer of the hybrid expert layer, and perform word segmentation processing based on the expert distribution probability of the deep layer of the hybrid expert layer.
7. A load balancing adjustment device based on a hybrid expert model, characterized in that, include: The predictive expert fingerprint acquisition module is used to acquire the predictive expert fingerprint of each waiting request in the waiting queue; wherein, the predictive expert fingerprint reflects the expert distribution probability of the waiting request in the hybrid expert model; The semantic load category acquisition module is used to acquire the semantic load category based on the accurate expert fingerprint of each historical running request, and to acquire the semantic load category corresponding to each waiting request based on the predicted expert fingerprint of each waiting request; wherein, the accurate expert fingerprint reflects the expert distribution result of the historical running requests in the hybrid expert model; The load balancing adjustment execution module is used to obtain the expected expert load of the hybrid expert model based on the clustering expert fingerprint of the semantic load category corresponding to each waiting request, so as to adjust the load of the hybrid expert model according to the expected expert load.
8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor, which enables the at least one processor to perform the load balancing adjustment method of the hybrid expert model according to any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the load balancing adjustment method of the hybrid expert model according to any one of claims 1-6.
10. A computer program product comprising a computer program that, when executed by a processor, implements the load balancing adjustment method of the hybrid expert model according to any one of claims 1-6.
