A Load Balancing Method, Device and Electronic Device for a Mixture of Experts Model

Abstract

The present application provides a load balancing method, device and electronic device for a mixture of experts model. The load balancing method is applied in the training stage of the mixture of experts model and includes: obtaining the number of pending loads of each sub-model under each computing node; for each sub-model, summing up the number of pending loads of the sub-model under each computing node to obtain the total actual load received by the sub-model, and determining the total preset load of the sub-model under multiple computing nodes; based on the total actual load and the total preset load of each sub-model, determining the pending loads to be discarded from the pending loads, and discarding the pending loads to enable each sub-model in the mixture of experts model to achieve load balancing. Through the method and device, only one global communication synchronization is required to significantly reduce the number of discarded loads, without affecting the representation ability of the mixture of experts model, and at the same time preventing the waste of computing power of the mixture of experts model.

Description

Technical Field

[0001] This application relates to the technical field of mixture-of-experts models, and particularly to a load balancing method, device, and electronic device for a mixture-of-experts model. Background Art

[0002] In recent years, deep neural networks centered around the Transformer architecture have achieved breakthrough progress. In this context, distributed training technologies (such as the mixture-of-experts model) have become the key technical path for efficient training of ultra-large-scale models.

[0003] In the mixture-of-experts model architecture, there are inherent threshold constraints on the parallel processing capabilities of each expert sub-model. When the local load distribution is unbalanced, the conventional local discarding method directly discards the load exceeding the processing upper limit of the sub-model. If too much load is discarded on the training side, it will lead to insufficient update of sub-model parameters, reduce the model representation ability, and also cause serious computing power loss. If the load distribution is more uneven, the number of discarded loads will be more, ultimately leading to deterioration of the overall model performance.

[0004] Therefore, in the training stage, how to construct a dynamic and adaptive expert load balancing mechanism to ensure that each sub-model obtains an equivalent optimization intensity in distributed training has become a technical problem that cannot be underestimated. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a load balancing method, device, and electronic device for a mixture-of-experts model, which determine the to-be-discarded load from the to-be-processed load according to the actual total load of each sub-model and the preset total load, so as to achieve load balancing for each sub-model in the mixture-of-experts model. Only one global communication synchronization is required to significantly reduce the number of discarded loads. The number of discarded loads is low, which does not affect the representation ability of the mixture-of-experts model, and at the same time prevents the waste of computing power of the mixture-of-experts model.

[0006] In a first aspect, an embodiment of this application provides a load balancing method for a mixture-of-experts model, which is applied to the training stage of the mixture-of-experts model. The load balancing method includes:

[0007] Obtain the number of to-be-processed loads of each sub-model under each computing node;

[0008] For each sub-model, sum up the number of to-be-processed loads of this sub-model under each computing node to obtain the actual total load received by this sub-model, and determine the preset total load of this sub-model under multiple computing nodes;

[0009] Determine the load to be discarded from the to-be-processed load based on the total actual load of each sub-model and the preset total load, and discard the load to be discarded, so as to achieve load balancing for each sub-model in the mixture-of-experts model.

[0010] Further, the determining the load to be discarded from the to-be-processed load based on the total actual load of each sub-model and the preset total load includes:

[0011] For each sub-model, determine whether the total actual load of this sub-model is greater than the preset total load of this sub-model;

[0012] If so, determine this sub-model as the target sub-model, and calculate the difference between the total actual load and the preset total load of the target sub-model;

[0013] Based on the difference corresponding to the target sub-model, determine the load to be discarded from the to-be-processed load of the target sub-model under each computing node;

[0014] Or,

[0015] For each sub-model, determine whether the total actual load of this sub-model is greater than the preset total load of this sub-model;

[0016] If so, determine this sub-model as the target sub-model, and calculate the load difference ratio based on the difference between the total actual load and the preset total load of the target sub-model; wherein, the load difference ratio is the ratio of the difference to the preset total load;

[0017] Based on the load difference ratio corresponding to the target sub-model, determine the load to be discarded from the to-be-processed load of the target sub-model under each computing node.

[0018] Further, the determining the load to be discarded from the to-be-processed load of the target sub-model under each computing node based on the difference corresponding to the target sub-model includes:

[0019] Determine the first discard quantity based on the ratio of the difference to the number of computing nodes;

[0020] For each computing node, based on the first discard quantity, determine the load to be discarded corresponding to the first discard quantity from the to-be-processed load of the target sub-model under this computing node.

[0021] Further, the determining the load to be discarded from the to-be-processed load of the target sub-model under each computing node based on the difference corresponding to the target sub-model includes:

[0022] Based on the difference value, determine the second discard quantity corresponding to each computing node; wherein, the sum of the second discard quantities of each computing node is equal to the difference value.

[0023] For each computing node, based on the second discard quantity corresponding to the computing node, determine the discarded load corresponding to the second discard quantity from the to-be-processed load of the target sub-model under the computing node.

[0024] Further, the discarded load determined from the to-be-processed load of the target sub-model under each computing node based on the load difference ratio corresponding to the target sub-model includes:

[0025] For each computing node, determine the third discard quantity corresponding to the computing node based on the product of the quantity of the to-be-processed load of the target sub-model under the computing node and the load difference ratio.

[0026] Based on the third discard quantity corresponding to the computing node, determine the discarded load corresponding to the third discard quantity from the to-be-processed load of the target sub-model under the computing node.

[0027] Further, the load balancing method further includes:

[0028] When it is detected that there is a sub-model with the actual total load greater than the preset total load, obtain the score corresponding to each to-be-processed load; wherein, the score represents the matching degree between each to-be-processed load and the sub-model corresponding to the to-be-processed load, and the higher the score, the higher the matching degree.

[0029] Sort the multiple to-be-processed loads in ascending order based on the scores, and determine the first preset quantity of to-be-processed loads from the sorting result as the discarded load.

[0030] Further, the load balancing method further includes:

[0031] When it is detected that there is a sub-model with the actual total load greater than the preset total load, determine the second preset quantity of to-be-processed loads from the to-be-processed loads of multiple sub-models under multiple computing nodes as the discarded load.

[0032] Further, the obtaining of the quantity of the to-be-processed load of each sub-model under each computing node includes:

[0033] The hybrid expert model includes a communication layer, the communication layer sends a acquisition request to each computing node, and each computing node returns meta-information including the quantity of the to-be-processed load to the communication layer.

[0034] Second aspect, an embodiment of the present application further provides a load balancing device for a mixture of experts model, which is applied to the training stage of the mixture of experts model. The load balancing device includes:

[0035] A load quantity acquisition module, configured to acquire the quantity of the pending loads of each sub-model under each computing node;

[0036] A total load calculation module, configured to, for each sub-model, sum up the quantities of the pending loads of the sub-model under each computing node to obtain the actual total load received by the sub-model, and determine the preset total load of the sub-model under multiple computing nodes;

[0037] A first discarded load determination module, configured to determine discarded loads from the pending loads based on the actual total load and the preset total load of each sub-model, and discard the discarded loads, so that each sub-model in the mixture of experts model achieves load balance.

[0038] Third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the load balancing method of the mixture of experts model as described above are executed.

[0039] A load balancing method, a device, and an electronic device for a mixture of experts model provided by an embodiment of the present application. The load balancing method is applied to the training stage of the mixture of experts model. First, the quantity of the pending loads of each sub-model under each computing node is acquired; then, for each sub-model, the quantities of the pending loads of the sub-model under each computing node are summed up to obtain the actual total load received by the sub-model, and the preset total load of the sub-model under multiple computing nodes is determined; finally, based on the actual total load and the preset total load of each sub-model, discarded loads are determined from the pending loads, and the discarded loads are discarded, so that each sub-model in the mixture of experts model achieves load balance.

[0040] During the training of the mixture of experts model in the present application, the quantities of the loads received by each sub-model are counted, and the discarded loads are determined from the pending loads according to the actual total load and the preset total load of each sub-model and discarded, so that each sub-model in the mixture of experts model achieves load balance. According to the load balancing method provided by the present application, only one global communication synchronization is required to significantly reduce the quantity of discarded loads. The quantity of discarded loads is low, which will not affect the representation ability of the mixture of experts model, and at the same time, it prevents the waste of computing power of the mixture of experts model.

[0041] To make the above objects, features, and advantages of the present application more obvious and understandable, the following provides preferred embodiments and detailed descriptions in conjunction with the accompanying drawings as follows. Description of the Drawings

[0042] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative efforts.

[0043] Figure 1 A flowchart of a load balancing method for a mixture-of-experts model provided by an embodiment of the present application;

[0044] Figure 2 A schematic structural diagram of a load balancing device for a mixture-of-experts model provided by an embodiment of the present application;

[0045] Figure 3 A schematic structural diagram of a load balancing device for a mixture-of-experts model provided by an embodiment of the present application;

[0046] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application that is required to be protected, but only represents the selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by those of ordinary skill in the art without creative efforts belongs to the scope of protection of the present application.

[0048] First, the applicable application scenarios of the present application will be introduced. The present application can be applied to the technical field of mixture-of-experts models.

[0049] In recent years, deep neural networks centered on the Transformer architecture have achieved breakthrough progress. In this context, distributed training technologies (such as the mixture-of-experts model) have become the key technical paths to achieve efficient training of ultra-large-scale models.

[0050] It has been found through research that in the architecture of the mixture of experts model, there are inherent threshold constraints on the parallel processing capabilities of each expert sub-model. When the local load distribution is unbalanced, the conventional local discarding method directly discards the load that exceeds the processing upper limit of the sub-model. If too much load is discarded on the training side, it will lead to insufficient update of the sub-model parameters, reduce the model representation ability, and at the same time cause relatively serious computing power loss. If the load distribution is more uneven, the number of discarded loads will be more, ultimately leading to the deterioration of the performance of the overall model.

[0051] Therefore, in the training stage, how to construct a dynamic adaptive expert load balancing mechanism to ensure that each sub-model obtains an equivalent optimization intensity in distributed training has become a technical problem that cannot be underestimated.

[0052] Based on this, the embodiments of the present application provide a load balancing method for a mixture of experts model, which can significantly reduce the number of discarded loads with only one global communication synchronization. The number of discarded loads is low and does not affect the representation ability of the mixture of experts model, while also preventing the waste of computing power of the mixture of experts model.

[0053] Please refer to Figure 1 , Figure 1 , which is a flowchart of a load balancing method for a mixture of experts model provided by the embodiments of the present application. The load balancing method provided by the embodiments of the present application is applied to the training stage of the mixture of experts model. As shown in Figure 1 , the load balancing method includes:

[0054] S101, obtain the number of pending loads of each sub-model under each computing node.

[0055] It should be noted that in distributed computing, a computing node usually represents an independent process or computing unit responsible for executing a part of the computing tasks of the mixture of experts model. Each computing node has multiple sub-models to execute corresponding computing tasks. The sub-model is the expert model in the mixture of experts model. Each expert model is an independent neural network (such as a fully connected layer, a Transformer module, etc.) responsible for processing inputs of specific patterns or features.

[0056] For the above step S101, in specific implementation, obtain the number of pending loads of each sub-model in the mixture-of-experts model under each computing node. Here, the mixture-of-experts model includes a gating module and softmax. The gating module and softmax cooperate with each other to determine which sub-model the obtained each pending load is assigned to. Among them, the gating module is used to generate the assignment probability of each pending load for each sub-model, and softmax normalizes the above assignment probabilities, selects the model with the highest assignment probability to assign the current pending load. After all pending loads are assigned, obtain the assignment results of all pending loads to form the number of pending loads of each sub-model under each computing node in this application.

[0057] As an optional embodiment, for the above step S101, the obtaining the number of pending loads of each sub-model under each computing node includes:

[0058] The mixture-of-experts model includes a communication layer. The communication layer sends a fetch request to each computing node, and each computing node returns meta-information containing the number of pending loads to the communication layer.

[0059] Here, in the distributed implementation of the mixture-of-experts model, the interaction of the communication layer, computing nodes, and load meta-information is the key to achieving efficient task scheduling and load balancing. The communication layer coordinates data transmission between different computing nodes in the mixture-of-experts model and obtains the number of pending loads of each computing node in real time. The meta-information includes the number of loads currently queuing for processing in each computing node.

[0060] For the above step, in specific implementation, the communication layer in the mixture-of-experts model is responsible for obtaining the number of pending loads. The communication layer sends a fetch request to all computing nodes at a fixed time interval. After receiving the fetch request, the computing node returns meta-information containing the number of pending loads to the communication layer based on the fetch request. Here, this application takes the mixture-of-experts model including two computing nodes and five sub-models as an example for explanation. For example, if the meta-information containing the number of pending loads returned by each computing node to the communication layer is [6, 6, 9, 15, 14], [13, 13, 9, 8, 7], then the number of pending loads of each sub-model under each computing node can be determined according to the above meta-information. Specifically, the number of pending loads of the first sub-model under the first computing node is 6, and the number of pending loads of the first sub-model under the second computing node is 13; the number of pending loads of the second sub-model under the first computing node is 6, and the number of pending loads of the second sub-model under the second computing node is 13, and so on. Here, it should be noted that the above load distribution example is only for easy understanding, and the actual load is at least in the millions or even hundreds of millions level.

[0061] S102. For each sub-model, sum up the number of pending loads of the sub-model under each computing node to obtain the total actual load received by the sub-model, and determine the total preset load of the sub-model under multiple computing nodes.

[0062] Here, the total preset load refers to the upper limit of the load that the sub-model can process under multiple computing nodes. The total actual load refers to the total load actually received by the sub-model under multiple computing nodes.

[0063] Regarding the above step S102, in specific implementation, for each sub-model, sum up the number of pending loads of the sub-model under each computing node to obtain the total actual load received by the sub-model. Specifically, continuing with the example in the above step S101, when the number of pending loads of the first sub-model under the first computing node is 6 and the number of pending loads of the first sub-model under the second computing node is 13, add the two numbers to determine that the total actual load received by the sub-model is 19. Then, determine the preset load quantity of the sub-model under each computing node, and add up the preset load quantities under each computing node to obtain the total preset load of the sub-model under multiple computing nodes. Here, as an example, when the preset load quantity of the sub-model under each computing node is 10 and there are two computing nodes, it can be determined that the total preset load of the sub-model under multiple computing nodes is 20.

[0064] S103. Based on the total actual load and the total preset load of each sub-model, determine the loads to be discarded from the pending loads, and discard the loads to be discarded, so that each sub-model in the mixture-of-experts model achieves load balancing.

[0065] Regarding the above step S103, in specific implementation, after determining the total actual load and the total preset load of each sub-model in the above step S102, based on the total actual load and the total preset load of each sub-model, determine the loads to be discarded that need to be discarded from the pending loads received by the sub-model. Discarding the determined loads to be discarded can make each sub-model in the mixture-of-experts model achieve load balancing.

[0066] Here, in the load balancing method provided in the embodiments of the present application, two ways are provided to determine the loads to be discarded from the pending loads. The first is to determine the loads to be discarded according to the difference between the total actual load and the total preset load, and the second is to determine the loads to be discarded according to the load difference ratio between the total actual load and the total preset load.

[0067] First, an explanation is given for the first method, which determines the load to be discarded based on the difference between the total actual load and the total preset load. Specifically, for the above-mentioned step S103, determining the load to be discarded from the loads to be processed based on the total actual load and the total preset load of each sub-model includes:

[0068] I: For each sub-model, determine whether the total actual load of the sub-model is greater than the total preset load of the sub-model.

[0069] II: If so, determine the sub-model as the target sub-model, and calculate the difference between the total actual load and the total preset load of the target sub-model.

[0070] For the above steps I - II, in specific implementation, for each sub-model, first determine whether the total actual load of the sub-model is greater than the total preset load of the sub-model. If the total actual load of the sub-model is less than or equal to the total preset load of the sub-model, it can be considered that the sub-model can process the loads it receives. If the total actual load of the sub-model is greater than the total preset load of the sub-model, it is considered that the loads received by the sub-model exceed its processing limit, and load discarding is required to ensure the load balance of the sub-model. At this time, the sub-model is used as the target sub-model, and the difference between the total actual load and the total preset load of the target sub-model is calculated. Continuing the above example, according to the number of loads to be processed of each sub-model under each computing node obtained, the total actual load of the fourth sub-model is determined to be 23, and the total actual load of the fifth sub-model is determined to be 21, both of which exceed the total preset load of 20. Then, the fourth sub-model is determined as the first target sub-model, and the fifth sub-model is determined as the second target sub-model. The difference corresponding to the first target sub-model is 3, and the difference corresponding to the second target sub-model is 1.

[0071] III: Based on the difference corresponding to the target sub-model, determine the load to be discarded from the loads to be processed of the target sub-model under each computing node.

[0072] For the above step III, in specific implementation, after determining the difference between the total actual load and the total preset load of the target sub-model, based on this difference, determine the load to be discarded from the loads to be processed of the target sub-model under each computing node.

[0073] According to the load balancing method provided by the present application, the loads to be processed can be evenly discarded according to the calculated difference. As an optional embodiment, for the above step III, in specific implementation, the determining the load to be discarded from the loads to be processed of the target sub-model under each computing node based on the difference corresponding to the target sub-model includes:

[0074] i: Determine a first discard quantity based on the ratio of the difference value to the number of computing nodes.

[0075] Regarding step i above, in specific implementation, determine the first discard quantity based on the ratio between the difference value corresponding to the target sub-model and the number of multiple computing nodes. Here, continuing the example of the above steps, for the first target sub-model, the difference value is 3, and the ratio of the difference value 3 to the number of computing nodes 2 is 1.5. When it cannot be divided evenly, round up, that is, the determined first discard quantity corresponding to the first target sub-model is 2. For the second target sub-model, the difference value is 1, and the ratio of the difference value to the number of computing nodes is 0.5. When it cannot be divided evenly, round up, that is, the determined first discard quantity corresponding to the second target sub-model is 1.

[0076] ii: For each computing node, determine the discard load corresponding to the first discard quantity from the pending load of the target sub-model under this computing node based on the first discard quantity.

[0077] Regarding step ii above, in specific implementation, for each computing node, determine the discard load corresponding to the first discard quantity from the pending load of the target sub-model under this computing node based on the first discard quantity determined in step i above. Here, continuing the above example, for the first target sub-model, the first discard quantity is 2. Therefore, two discard loads are determined under each computing node, that is, two pending loads are discarded from the pending load of the first target sub-model under each computing node. For the second target sub-model, the first discard quantity is 1. Therefore, one discard load is determined under each computing node, that is, one pending load is discarded from the pending load of the second target sub-model under each computing node. After discarding, the meta-information of the number of pending loads of each computing node is [6, 6, 9, 13, 13], [13, 13, 9, 6, 6]. Through this discard method, a total of 6 loads are discarded, and this discard quantity is acceptable in the training phase. This method does not require additional communication.

[0078] According to the load balancing method provided by the present application, the pending load can also be selectively discarded or randomly discarded according to the calculated difference value. As an optional embodiment, regarding step III above, in specific implementation, determining the discard load from the pending load of the target sub-model under each computing node based on the difference value corresponding to the target sub-model includes:

[0079] (1) Determine a second discard quantity corresponding to each computing node based on the difference value.

[0080] For the above step (1), in specific implementation, based on the difference corresponding to the target sub-model, determine the second discard quantity corresponding to each computing node. Here, the sum of the second discard quantities of each computing node is equal to the difference. Continuing the example of the above step, for the first target sub-model, the difference is 3. Then, 3 loads can be randomly discarded in one computing node, or one computing node can be randomly selected to discard 1 load, and another computing node can discard 2 loads, or the computing node with a larger number of loads can be selected to discard 3 loads, as long as the total number of discarded loads is equal to the difference of 3. For the second target sub-model, the difference is 1. One load can be randomly discarded in one computing node, or the computing node with a larger number of loads can be selected to discard 1 load, as long as the total number of discarded loads is equal to the difference of 1.

[0081] (2) For each computing node, based on the second discard quantity corresponding to the computing node, determine the discarded loads corresponding to the second discard quantity from the loads to be processed by the target sub-model under the computing node.

[0082] For the above step (2), in specific implementation, for each computing node, based on the second discard quantity corresponding to the computing node, determine the discarded loads corresponding to the second discard quantity from the loads to be processed by the target sub-model under the computing node. Continuing the example in the above step, for the first target sub-model, 2 loads are discarded from the loads to be processed under the first computing node, and 1 load is discarded from the loads to be processed under the second computing node. For the second target sub-model, 1 load is discarded from the loads to be processed under the first computing node. After discarding, the meta-information of the number of loads to be processed for each computing node is [6, 6, 9, 13, 13], [13, 13, 9, 7, 7]. Through this discard method, a total of 4 loads are discarded, and the number of discarded loads is the lowest.

[0083] Here, the second method is further explained. The discarded loads are determined according to the load difference ratio between the actual total load and the preset total load. Specifically, for the above step S103, the determining of the discarded loads from the loads to be processed based on the actual total load and the preset total load of each sub-model includes:

[0084] A: For each sub-model, determine whether the actual total load of the sub-model is greater than the preset total load of the sub-model.

[0085] B: If so, determine the sub-model as the target sub-model, and calculate the load difference ratio based on the difference between the actual total load and the preset total load of the target sub-model.

[0086] For the above steps A - B, in specific implementation, for each sub - model, it is determined whether the total actual load of the sub - model is greater than the preset total load of the sub - model. If it is greater, it is considered that the load received by the sub - model exceeds its processing limit. At this time, the sub - model is determined as the target sub - model, and the load difference ratio is calculated based on the difference between the total actual load and the preset total load of the target sub - model. Specifically, the load difference ratio is the ratio of the difference to the preset total load. Here, continuing the example in step II above, the difference corresponding to the first target sub - model is 3, and the load difference ratio is 0.15. The difference corresponding to the second target sub - model is 1, and the load difference ratio is 0.05.

[0087] C: The load to be discarded determined from the loads to be processed of the target sub - model under each computing node based on the load difference ratio corresponding to the target sub - model.

[0088] For the above step C, in specific implementation, after determining the load difference ratio corresponding to the target sub - model, based on this load difference ratio, the load to be discarded is determined from the loads to be processed of the target sub - model under each computing node.

[0089] According to the load balancing method provided by this application, proportional discarding can be performed according to the calculated load difference ratio. As an optional embodiment, for the above step C, in specific implementation, the load to be discarded determined from the loads to be processed of the target sub - model under each computing node based on the load difference ratio corresponding to the target sub - model includes:

[0090] a: For each computing node, based on the product of the number of loads to be processed of the target sub - model under this computing node and the load difference ratio, the corresponding third discard quantity of this computing node is determined.

[0091] For the above step a, in specific implementation, for each computing node, based on the product of the number of loads to be processed of the target sub - model under this computing node and the load difference ratio, the corresponding third discard quantity of this computing node is determined. Here, continuing the example in the above steps, the load difference ratio of the first target sub - model is 0.15. Therefore, under the first computing node, 15 * 0.15 = 2.25, and rounding up, the third discard quantity is 3. Under the second computing node, 8 * 0.15 = 1.2, and rounding up, the third discard quantity is 2. For the second target sub - model, by the same calculation, the third discard quantity is 1 under both the first computing node and the second computing node.

[0092] b: Based on the corresponding third discard quantity of this computing node, the load to be discarded corresponding to the third discard quantity is determined from the loads to be processed of the target sub - model under this computing node.

[0093] For step b above, in specific implementation, based on the third discard quantity corresponding to the computing node, the to-be-discarded load corresponding to the third discard quantity is determined from the to-be-processed load of the target sub-model under this computing node. Here, continuing with the example in the above steps, for the first target sub-model, 3 loads are discarded under the first computing node and 2 loads are discarded under the second computing node. For the second target sub-model, 1 load is discarded under each computing node. After discarding, the meta-information of the to-be-processed load quantity of each computing node is [6, 6, 9, 12, 13], [13, 13, 9, 6, 6]. Through this discarding method, a total of 7 loads are discarded, and this discard quantity is acceptable in the training phase. This method does not require additional communication.

[0094] According to the load balancing method provided by this application, when the number of loads received by a sub-model is excessive, loads with lower scores can be selected for discarding. As an optional embodiment, the load balancing method provided by the embodiments of this application further includes:

[0095] When it is detected that there is a sub-model with the actual total load greater than the preset total load, obtain the score corresponding to each to-be-processed load; sort the multiple to-be-processed loads in ascending order based on the scores, and determine the first preset number of to-be-processed loads from the sorting result as the to-be-discarded loads.

[0096] Among them, the score represents the matching degree between each to-be-processed load and the sub-model corresponding to the to-be-processed load. The higher the score, the higher the matching degree.

[0097] For the above two steps, in specific implementation, when it is detected that there is a sub-model with the actual total load greater than the preset total load, obtain the score corresponding to each to-be-processed load. Here, the score corresponding to the to-be-processed load can be the score obtained by an expert scoring each to-be-processed load that needs to be processed, and this application does not make specific limitations on this. Then, sort the multiple to-be-processed loads in ascending order based on the obtained scores, and determine the first preset number of to-be-processed loads from the sorting result as the to-be-discarded loads.

[0098] According to the load balancing method provided by this application, when the number of loads received by a sub-model is excessive, random discarding of loads can be performed. As an optional embodiment, the load balancing method further includes:

[0099] When it is detected that there is a sub-model with the actual total load greater than the preset total load, determine the second preset number of to-be-processed loads from the to-be-processed loads of multiple sub-models under multiple computing nodes as the to-be-discarded loads.

[0100] For the above steps, in specific implementation, when it is detected that there is a sub-model with the total actual load greater than the total preset load, determine a second preset number of pending loads as the pending discarded loads from the pending loads of multiple sub-models under multiple computing nodes.

[0101] The load balancing method for the mixture of experts model provided by the embodiments of the present application is applied to the training phase of the mixture of experts model. First, obtain the number of pending loads of each sub-model under each computing node; then, for each sub-model, sum up the number of pending loads of the sub-model under each computing node to obtain the total actual load received by the sub-model, and determine the total preset load of the sub-model under multiple computing nodes; finally, based on the total actual load and the total preset load of each sub-model, determine the pending discarded loads from the pending loads, and discard the pending discarded loads to make each sub-model in the mixture of experts model achieve load balance.

[0102] During the training of the mixture of experts model in the present application, the number of loads received by each sub-model is counted, and the pending discarded loads are determined and discarded from the pending loads according to the total actual load and the total preset load of each sub-model, so that each sub-model in the mixture of experts model achieves load balance. According to the load balancing method provided by the present application, only one global communication synchronization is required to significantly reduce the number of discarded loads. The number of discarded loads is low, which will not affect the representation ability of the mixture of experts model, and at the same time prevents the waste of computing power of the mixture of experts model.

[0103] Please refer to Figure 2 、 Figure 3 , Figure 2 which is one of the structural schematic diagrams of a load balancing device for a mixture of experts model provided by the embodiments of the present application. Figure 3 which is the second structural schematic diagram of a load balancing device for a mixture of experts model provided by the embodiments of the present application. As shown in Figure 3 the load balancing device 200 includes:

[0104] A load quantity acquisition module 201, configured to obtain the number of pending loads of each sub-model under each computing node;

[0105] A total load calculation module 202, configured to, for each sub-model, sum up the number of pending loads of the sub-model under each computing node to obtain the total actual load received by the sub-model, and determine the total preset load of the sub-model under multiple computing nodes;

[0106] The first load-to-be-discarded determination module 203 is configured to determine the load to be discarded from the to-be-processed load based on the total actual load of each sub-model and the preset total load, and discard the load to be discarded, so as to achieve load balancing for each sub-model in the mixture-of-experts model.

[0107] Further, when the first load-to-be-discarded determination module 203 is configured to determine the load to be discarded from the to-be-processed load based on the total actual load of each sub-model and the preset total load, the first load-to-be-discarded determination module 203 is further configured to:

[0108] For each sub-model, determine whether the total actual load of the sub-model is greater than the preset total load of the sub-model;

[0109] If so, determine the sub-model as the target sub-model, and calculate the difference between the total actual load and the preset total load of the target sub-model;

[0110] Based on the difference corresponding to the target sub-model, determine the load to be discarded from the to-be-processed load of the target sub-model under each computing node;

[0111] Or,

[0112] For each sub-model, determine whether the total actual load of the sub-model is greater than the preset total load of the sub-model;

[0113] If so, determine the sub-model as the target sub-model, and calculate the load difference ratio based on the difference between the total actual load and the preset total load of the target sub-model; wherein, the load difference ratio is the ratio of the difference to the preset total load;

[0114] Based on the load difference ratio corresponding to the target sub-model, determine the load to be discarded from the to-be-processed load of the target sub-model under each computing node.

[0115] Further, when the first load-to-be-discarded determination module 203 is configured to determine the load to be discarded from the to-be-processed load of the target sub-model under each computing node based on the difference corresponding to the target sub-model, the first load-to-be-discarded determination module 203 is further configured to:

[0116] Determine the first discard quantity based on the ratio of the difference to the number of computing nodes;

[0117] For each computing node, based on the first discard quantity, determine the load to be discarded corresponding to the first discard quantity from the to-be-processed load of the target sub-model under the computing node.

[0118] Further, when the first discarded load determination module 203 is used to determine the discarded load from the pending loads of the target submodel under each computing node based on the difference corresponding to the target submodel, the first discarded load determination module 203 is further used for:

[0119] Based on the difference, determine the second discard quantity corresponding to each computing node; wherein, the sum of the second discard quantities of each computing node is equal to the difference;

[0120] For each computing node, based on the second discard quantity corresponding to the computing node, determine the discarded load corresponding to the second discard quantity from the pending loads of the target submodel under the computing node.

[0121] Further, when the first discarded load determination module 203 is used to determine the discarded load from the pending loads of the target submodel under each computing node based on the load difference ratio corresponding to the target submodel, the first discarded load determination module 203 is further used for:

[0122] For each computing node, based on the product of the quantity of the pending loads of the target submodel under the computing node and the load difference ratio, determine the third discard quantity corresponding to the computing node;

[0123] Based on the third discard quantity corresponding to the computing node, determine the discarded load corresponding to the third discard quantity from the pending loads of the target submodel under the computing node.

[0124] Further, as Figure 3 shown, the load balancing device 200 further includes a second discarded load determination module 204, and the second discarded load determination module 204 is used for:

[0125] When it is detected that there is a submodel with the actual total load greater than the preset total load, obtain the score corresponding to each pending load; wherein, the score represents the matching degree between each pending load and the submodel corresponding to the pending load, and the higher the score, the higher the matching degree;

[0126] Sort the multiple pending loads in ascending order based on the scores, and determine the first preset number of pending loads from the sorting result as the discarded loads.

[0127] Further, as Figure 3 shown, the load balancing device 200 further includes a third discarded load determination module 205, and the second discarded load determination module 205 is used for:

[0128] When a sub - model with the total number of actual loads greater than the total number of preset loads is detected, determine a second preset number of pending loads from the pending loads of multiple sub - models under multiple computing nodes as the pending discarded loads.

[0129] Further, when the load quantity acquisition module 201 is used to acquire the quantity of pending loads of each sub - model under each computing node, the load quantity acquisition module 201 is further used for:

[0130] The hybrid expert model includes a communication layer. The communication layer sends acquisition requests to each computing node, and each computing node returns meta - information containing the quantity of pending loads to the communication layer.

[0131] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As Figure 4 shown in, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0132] The memory 420 stores machine - readable instructions executable by the processor 410. When the electronic device 400 runs, the processor 410 communicates with the memory 420 through the bus 430. When the machine - readable instructions are executed by the processor 410, the steps of the load balancing method of the hybrid expert model in the method embodiment as described above can be executed. The specific implementation manner can refer to the method embodiment and will not be elaborated here. Figure 1 shown in.

[0133] An embodiment of the present application further provides a computer - readable storage medium. A computer program is stored on the computer - readable storage medium. When the computer program is run by a processor, the steps of the load balancing method of the hybrid expert model in the method embodiment as described above can be executed. The specific implementation manner can refer to the method embodiment and will not be elaborated here. Figure 1 shown in.

[0134] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above - described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated here.

[0135] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical functional division. In actual implementation, there may be other division methods. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some communication interfaces. The indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0138] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0139] Finally, it should be noted that the above-described embodiments are only specific implementation manners of the present application, used to illustrate the technical solutions of the present application, rather than limiting it. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: Any person skilled in the art within the technical scope disclosed by the present application can still modify the technical solutions recorded in the foregoing embodiments or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A load balancing method for a hybrid expert model, applied in the hybrid expert model training phase, characterized in that: The hybrid expert model includes a gating module and a softmax module, and the load balancing method includes: Based on the gating module and softmax, the sub-models for all pending loads are determined. Based on the distribution results of all pending loads, the number of pending loads for each sub-model under each computing node is obtained. For each sub-model, sum the number of loads to be processed by the sub-model at each computing node to obtain the total actual load received by the sub-model, and determine the total preset load of the sub-model at multiple computing nodes; Based on the total actual load of each sub-model and the total preset load, determining the load to be discarded from the load to be processed, and discarding the load to be discarded, so that each sub-model in the hybrid expert model achieves load balance; The step of determining the load to be discarded from the load to be processed based on the total actual load of each sub-model and the total preset load includes: For each sub-model, determine whether the total actual load of the sub-model is greater than the total preset load of the sub-model; If yes, the sub-model is determined as the target sub-model, and the difference between the actual total load of the target sub-model and the preset total load is calculated; Based on the difference value corresponding to the target sub-model, determining the load to be discarded from the load to be processed under each computing node of the target sub-model; or, For each sub-model, determine whether the total actual load of the sub-model is greater than the total preset load of the sub-model; If so, the submodel is determined as the target submodel, and a load difference ratio is calculated based on the difference between the actual total load of the target submodel and the preset total load; wherein the load difference ratio is the ratio between the difference and the preset total load; Based on the load difference ratio corresponding to the target sub-model, the load to be discarded is determined from the load to be processed under each computing node of the target sub-model.

2. The load balancing method according to claim 1, wherein: The determining, based on the difference corresponding to the target sub-model, the load to be discarded from the load to be processed under each computing node of the target sub-model includes: Determining a first discard quantity based on a ratio of the difference to the number of computing nodes; For each computing node, based on the first discard quantity, a to-be-discarded load corresponding to the first discard quantity is determined from the to-be-processed load of the target sub-model under the computing node.

3. The load balancing method according to claim 1, wherein: The determining, based on the difference corresponding to the target sub-model, the load to be discarded from the load to be processed under each computing node of the target sub-model includes: Based on the difference, determining a second discard quantity corresponding to each computing node; wherein the sum of the second discard quantities of each computing node is equal to the difference; For each computing node, based on the second discard quantity corresponding to the computing node, the load to be discarded corresponding to the second discard quantity is determined from the load to be processed by the target sub-model under the computing node.

4. The load balancing method according to claim 1, wherein: The load to be discarded determined from the load to be processed under each computing node of the target sub-model based on the load difference ratio corresponding to the target sub-model includes: For each computing node, determining a third discard quantity corresponding to the computing node based on a product of the quantity of to-be-processed loads of the target submodel under the computing node and the load difference ratio; Based on the third discard quantity corresponding to the computing node, the to-be-discarded load corresponding to the third discard quantity is determined from the to-be-processed load of the target sub-model under the computing node.

5. The load balancing method according to claim 1, wherein: The load balancing method further includes: When it is detected that there is a sub-model in which the total number of actual loads is greater than the total number of preset loads, a score corresponding to each load to be processed is obtained; wherein the score represents the degree of matching between each load to be processed and the sub-model corresponding to the load to be processed, and a higher score indicates a higher degree of matching; The plurality of to-be-processed loads are sorted in order from low to high based on the scores, and a first preset number of to-be-processed loads are determined from the sorting results as to-be-discarded loads.

6. The load balancing method according to claim 1, wherein: The load balancing method further includes: When it is detected that there is a sub-model whose total actual load is greater than the total preset load, a second preset number of loads to be processed is determined from the loads to be processed of the multiple sub-models under the multiple computing nodes as the loads to be discarded.

7. The load balancing method according to claim 1, wherein: The obtaining of the amount of load to be processed of each sub-model under each computing node includes: The hybrid expert model includes a communication layer, which sends an acquisition request to each computing node, and each computing node returns meta information including the amount of load to be processed to the communication layer.

8. A load balancing device for a hybrid expert model, applied in the hybrid expert model training phase, characterized in that: The hybrid expert model includes a gating module and a softmax module, and the load balancing device includes: The load quantity acquisition module is used to determine the sub-models for each pending load distribution based on the gating module and softmax, and obtain the number of pending loads for each sub-model under each computing node based on the distribution results of all pending loads; A total load calculation module is used to sum the number of loads to be processed by each sub-model at each computing node to obtain the total actual load received by the sub-model and determine the total preset load of the sub-model at multiple computing nodes; a first to-be-discarded load determination module, configured to determine a to-be-discarded load from the to-be-processed load based on the total actual load of each sub-model and the total preset load, and to discard the to-be-discarded load so as to achieve load balancing for each sub-model in the hybrid expert model; When the first load-to-be-discarded determining module is used to determine the load to be discarded from the loads to be processed based on the total actual load and the total preset load of each sub-model, the first load-to-be-discarded determining module is further used to: For each sub-model, determine whether the total actual load of the sub-model is greater than the total preset load of the sub-model; If yes, the sub-model is determined as the target sub-model, and the difference between the actual total load of the target sub-model and the preset total load is calculated; Based on the difference value corresponding to the target sub-model, determining the load to be discarded from the load to be processed under each computing node of the target sub-model; or, For each sub-model, determine whether the total actual load of the sub-model is greater than the total preset load of the sub-model; If so, the submodel is determined as the target submodel, and a load difference ratio is calculated based on the difference between the actual total load of the target submodel and the preset total load; wherein the load difference ratio is the ratio between the difference and the preset total load; Based on the load difference ratio corresponding to the target sub-model, the load to be discarded is determined from the load to be processed under each computing node of the target sub-model.

9. An electronic device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus. When the processor runs the machine-readable instructions, the steps of the load balancing method of the hybrid expert model as described in any one of claims 1 to 7 are executed.

Images