Hybrid expert network training method, device, equipment and computer program product

Abstract

The invention discloses a hybrid expert network training method, device and equipment and a computer program product, experts in a hybrid expert layer are equally divided into a plurality of groups and equally distributed to a plurality of pieces of equipment, and different pieces of equipment share the same routing module. In the training process, for an input hidden layer sequence of the mixed expert layer, the score of each path of experts is calculated through a routing module, for each expert group, activation experts in the group are selected according to the scores of the experts in the group, the number of the activation experts in different groups is the same, and each expert group processes the complete hidden layer sequence. And processing the input hidden layer sequence by the activation experts in each expert group to obtain the output of the activation experts, and obtaining the output of the hybrid expert layer based on the output of each activation expert. According to the method and the device, absolute balance of calculation among the devices is ensured, and the effect of the model is not influenced by drop-pad (namely data discarding and 0 filling).

Description

Technical Field

[0001] The present application relates to the field of model training technology, and more specifically, to a hybrid expert network training method, device, equipment and computer program product. Background Art

[0002] In the past few years, large language models (LLMs) have experienced significant development, and many LLMs have begun to adopt a mixture of experts (MoE) network structure. The MoE structure dynamically activates a part of the experts through expert routing to replace the dense FFN layer. This mechanism enables computing resources to be concentrated on the most relevant parts and reduce unnecessary computing overhead. Compared with the traditional Dense model, the MoE model has a more efficient training speed.

[0003] Existing MOE technology usually uses the Expert Parallelism (EP) method in parallel training, assigning different experts to different computing devices (graphics cards) to achieve parallel computing. This method will introduce load balancing problems, that is, it cannot well guarantee the load balance between different computing devices. One load balancing solution in the prior art is to limit the maximum number of tokens that each expert can process by setting the expert capacity: when the number of routing tokens of an expert exceeds the preset upper limit, the calculation of the excess part is usually discarded to ensure load balancing; when the number of tokens processed by an expert is less than the preset value, the matrix is ​​usually filled with 0 to make the load as equal as possible. The above-mentioned prior art itself cannot achieve load balancing of different computing devices. When encountering uneven load, it is handled by drop-pad (i.e. discarding data and filling 0), which will seriously affect the effect of the model. Summary of the invention

[0004] In view of the above problems, this application is proposed to provide a hybrid expert network training method, device, equipment and computer program product to achieve load balancing between different devices without affecting the effect of the model. The specific solution is as follows:

[0005] In a first aspect, a hybrid expert network training method is provided. For a hybrid expert layer in a hybrid expert network to be trained, all experts therein are evenly divided into a plurality of groups, and each of the divided expert groups is evenly distributed to a plurality of devices, and different devices share the same routing module. The method includes:

[0006] Sending training data to the hybrid expert network to obtain a hidden layer sequence output by the previous layer of the hybrid expert layer as input to the hybrid expert layer;

[0007] The routing module calculates the score of each expert according to the hidden layer sequence, and for each expert group, selects activated experts in the group according to the scores of each expert in the group, and the number of activated experts in different groups is the same;

[0008] Inputting the hidden layer sequence into each of the expert groups respectively, and processing it by the activated experts in the expert group, so as to obtain the output of each activated expert in each of the expert groups;

[0009] Based on the output of each activated expert in each expert group, the output of the mixed expert layer is obtained, and is passed backward until the output of the network is obtained through the last layer. The loss function is calculated based on the output of the network and the true value label, and the network parameters are updated according to the loss function.

[0010] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of obtaining the output of the hybrid expert layer based on the output of each activated expert in each expert group includes:

[0011] The scores of all activated experts in each expert group are normalized, and the outputs of each activated expert are weightedly fused using the normalized weighted probability, and the weighted fusion result is used as the output of the hybrid expert layer.

[0012] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the hybrid expert network training process adopts a tensor parallel method, and the process of calculating the score of each expert according to the hidden layer sequence through the routing module includes:

[0013] Dividing the hidden layer sequence into a plurality of subsequences according to the tensor parallel size, and inputting each subsequence into a device respectively, wherein the tensor parallel size is consistent with the number of the devices;

[0014] On each device, the score of each expert is calculated based on the subsequence input by the current device through the shared routing module, and the subsequences input by other devices are obtained by an all-gather communication method to obtain the complete hidden layer sequence. The score of each expert calculated by other devices is obtained by an all-gather communication method to obtain the score of each expert in the complete hidden layer sequence.

[0015] In a possible design, in another implementation of the first aspect of the embodiments of the present application, a process of acquiring a subsequence input by another device using an all-gather communication mode includes:

[0016] Use asynchronous all-gather communication to obtain subsequences input from other devices;

[0017] The process of using the all-gather communication mode to obtain the scores of each expert calculated by other devices includes:

[0018] A synchronous all-gather communication method is used to obtain the scores of each expert calculated by other devices.

[0019] In a possible design, in another implementation of the first aspect of the embodiment of the present application, the loss function of the hybrid expert network training process further includes:

[0020] The global expert load balancing loss function is used to constrain the output distribution of the routing module so that the input hidden layer sequence can be evenly distributed to all experts.

[0021] In a second aspect, a hybrid expert network training device is provided. For a hybrid expert layer in a hybrid expert network to be trained, all experts therein are evenly divided into a plurality of groups, and each of the divided expert groups is evenly distributed to a plurality of devices, and different devices share the same routing module. The device includes:

[0022] A data input unit, used for sending training data into the hybrid expert network to obtain a hidden layer sequence output by a previous layer of the hybrid expert layer as input of the hybrid expert layer;

[0023] A first calculation unit is used to calculate the score of each expert according to the hidden layer sequence through the routing module, and for each expert group, select an activated expert in the group according to the score of each expert in the group, and the number of activated experts in different groups is the same;

[0024] A second computing unit is used to input the hidden layer sequence into each of the expert groups respectively, and the activated experts in the expert groups process the hidden layer sequence to obtain the output of each activated expert in each of the expert groups;

[0025] The third computing unit is used to obtain the output of the mixed expert layer based on the output of each activated expert in each expert group, and pass it backward until the output of the network is obtained through the last layer, calculate the loss function based on the output of the network and the true value label, and update the network parameters according to the loss function.

[0026] In a possible design, in another implementation of the second aspect of the embodiment of the present application, the process of obtaining the output of the hybrid expert layer by the third computing unit based on the output of each activated expert in each of the expert groups includes:

[0027] The scores of all activated experts in each expert group are normalized, and the outputs of each activated expert are weightedly fused using the normalized weighted probability, and the weighted fusion result is used as the output of the hybrid expert layer.

[0028] In a third aspect, an electronic device is provided, comprising: a memory and a processor;

[0029] The memory is used to store programs;

[0030] The processor is used to execute the program to implement the hybrid expert network training method described in any one of the first aspects of the present application.

[0031] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the hybrid expert network training method described in any one of the first aspects of the present application is implemented.

[0032] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the hybrid expert network training method described in any one of the aforementioned first aspects of the present application.

[0033] By means of the above technical solution, in order to solve the problem of unbalanced load in the training process of the hybrid expert network, the present application evenly divides the experts in the hybrid expert layer in the hybrid expert network into several groups, and evenly distributes the divided expert groups to several devices, and the same routing module is shared between different devices. During the training process, for the input hidden layer sequence of the hybrid expert layer, the score of each expert is calculated through the routing module, and for each expert group, the activated experts in the group are selected according to the scores of the experts in the group, and the number of activated experts in different groups is the same, and then each expert group processes the complete hidden layer sequence, and the activated experts in each expert group process the input hidden layer sequence to obtain the output of the activated experts, and based on the output of each activated expert, the output of the hybrid expert layer is obtained, which is passed backward until the last layer obtains the output of the network, and the loss function is calculated based on this for training. Obviously, the solution of the present application groups experts and distributes them evenly to different devices. Each expert group processes the complete hidden layer sequence, ensuring absolute balance of calculations between devices, thereby avoiding calculation waiting between different devices due to uneven loads, maximizing the utilization of hardware devices, and not affecting the effect of the model due to drop-pad (i.e. discarding data and filling in 0). BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0035] Figure 1 A schematic diagram of a hybrid expert network local structure and a hybrid expert layer training process provided in an embodiment of the present application;

[0036] Figure 2 A flow chart of a hybrid expert network training method provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of a data flow structure of a hybrid expert network training process provided in an embodiment of the present application;

[0038] Figure 4 A schematic diagram of the structure of a hybrid expert network training device provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] Large language models (LLMs) have experienced significant development in the past few years. Generally, as the number of parameters increases, the intelligence level of LLMs also increases, making them perform better in various tasks. However, this progress comes at the cost of consuming more computing resources for training. For a Transformer-based LLM, its feedforward neural network (FFN) layer usually occupies nearly 2 / 3 of the parameters of the entire model. When the model size continues to increase, the parameters of the FFN layer and its computational consumption also increase proportionally.

[0042] In order to break the coupled growth between computational cost and parameter quantity, many LLMs have recently begun to adopt a Mixture-of-Expert (MoE) network structure. This sparse expert network allows a significant increase in the number of model parameters while keeping the number of floating-point operations (FLOPs) required for calculation relatively unchanged. The MoE structure dynamically activates a part of the experts through expert routing to replace the dense FFN layer. This mechanism enables computing resources to be concentrated on the most relevant parts, reducing unnecessary computing overhead. Compared with the traditional Dense model, the MoE model has a more efficient training speed.

[0043] Existing MOE technology usually uses the Expert Parallelism (EP) method in parallel training, assigning different experts to different computing devices (graphics cards) to achieve parallel computing. This method will introduce load balancing problems, that is, it cannot well guarantee the load balance between different computing devices. One load balancing solution in the prior art is to limit the maximum number of tokens that each expert can process by setting the expert capacity: when the number of routing tokens of an expert exceeds the preset upper limit, the calculation of the excess part is usually discarded to ensure load balancing; when the number of tokens processed by an expert is less than the preset value, the matrix is ​​usually filled with 0 to make the load as equal as possible. The above-mentioned prior art itself cannot achieve load balancing of different computing devices. When encountering uneven load, it is handled by drop-pad (i.e. discarding data and filling 0), which will seriously affect the effect of the model.

[0044] Recent research on MOE shows that activating more small-sized experts is more advantageous than activating an equal number of large-sized experts. More and smaller experts increase the difficulty of load balancing. Existing technologies add more load balancing losses to alleviate load imbalance in order to constrain the load between experts and devices, but do not fundamentally solve the problem of load imbalance.

[0045] In view of this, the present application provides a hybrid expert network training method to achieve absolute balance of calculations between different devices. The hybrid expert network training method of the present application can be applied to a server, which can include several devices (such as graphics cards).

[0046] Combination Figure 1 As shown, for the mixed expert layer Grouped MoE Layer in the mixed expert network to be trained, all experts are evenly divided into several groups ( Figure 1The example is divided into expert group 1-expert group G, each expert group contains n experts, that is, FFN1-FFNn), and each expert group is evenly distributed to several devices, and different devices share the same routing module (Gate network). In other words, the parameters of the routing modules deployed on different devices are consistent.

[0047] In addition to the hybrid expert layer, the hybrid expert network also includes other network structures, such as Figure 1 Taking a transformer encoder block as an example, the self-attention module and the normalization layer Layer Norm can also be included before the hybrid expert layer. Similarly, the hybrid expert network can also include several layers of networks after the hybrid expert layer. Figure 1 Not fully shown.

[0048] Next, combine Figure 2 As shown, the hybrid expert network training method of this embodiment includes the following steps:

[0049] Step S100: Send the training data to the hybrid expert network to obtain the hidden layer sequence output by the previous layer of the hybrid expert layer as the input of the hybrid expert layer.

[0050] Specifically, the training data is sent into the hybrid expert network and processed by the layers before the hybrid expert layer to obtain the input of the hybrid expert layer, and the input of the hybrid expert layer is defined as the hidden layer sequence x.

[0051] Step S110: Calculate the score probs of each expert according to the hidden layer sequence through the routing module. For each expert group, select the activated experts in the group according to the scores of each expert in the group. The number of activated experts in different groups is the same.

[0052] Specifically, the routing module assigns expert scores to the hidden layer sequence x. In each expert group, according to the scores of each expert in the group, the top k / G activated experts are selected. Where k is the number of all activated experts. Since the number of activated experts in each expert group remains the same, the number of activated experts in each expert group is k / G. In this step, for each expert group, according to the scores of each expert in the group, the k / G experts with the highest scores are selected as activated experts.

[0053] Step S120: input the hidden layer sequence into each expert group respectively, and process it by the activated experts in the expert group to obtain the output of each activated expert in each expert group.

[0054] Specifically, in order to ensure the balance of computational load among different devices, the input of each expert group is a complete hidden layer sequence x. The activated experts in the expert group perform computational processing on the input hidden layer sequence x to obtain the output of each activated expert.

[0055] Since the number of expert groups on different devices is the same, the number of experts in the expert group and the number of activated experts are also the same, and the input of each expert group is also the same hidden layer sequence x, this can ensure absolute balance in calculations between different devices.

[0056] Step S130: Based on the output of each activated expert in each expert group, the output of the mixed expert layer is obtained, and it is passed backward until the output of the network is obtained through the last layer. The loss function is calculated based on the output of the network and the true value label, and the network parameters are updated according to the loss function.

[0057] Specifically, in this step, the outputs of each activated expert in each expert group can be fused to obtain the output of the mixed expert layer, and then the output is transmitted backward until the predicted output of the entire network is obtained through the last layer of the mixed expert network. The loss function is calculated based on the predicted output and the pre-acquired true value label, and the network parameters are updated according to the loss function.

[0058] It is understandable that the loss function of the hybrid expert network training process is not limited to the above loss function calculated based on the network output and the true value label. Other loss functions can also be designed to improve the training effect. Figure 1 As shown, after the routing module obtains the score of each expert (also called probability probs), the global expert load balancing loss L can also be calculated. aux , the global expert load balancing loss is mainly used to constrain the output distribution of the routing module, in order to allow the input hidden layer sequence x to be evenly distributed to all experts. The following example shows the global expert load balancing loss L aux A way to calculate:

[0059]

[0060]

[0061]

[0062] Among them, α is a hyperparameter that scales the Loss weight, E represents the total number of experts, i represents the expert number, K represents the number of activated experts, t represents the subscript of the token sequence, and p i,t It represents the proportion of token t assigned to expert i.

[0063] The above global expert load balancing loss L auxThe physical meaning of can be understood as that, for a complete hidden layer sequence, the distribution strategy of the routing module is such that the mean probability of tokens assigned to all experts is balanced.

[0064] The method provided in the embodiment of the present application, in order to solve the problem of unbalanced load in the training process of the hybrid expert network, evenly divides each expert in the hybrid expert layer in the hybrid expert network into several groups, and evenly distributes each expert group after division to several devices, and different devices share the same routing module. During the training process, for the input hidden layer sequence of the hybrid expert layer, the score of each expert is calculated through the routing module, and for each expert group, the activated experts in the group are selected according to the scores of the experts in the group. The number of activated experts in different groups is the same, and then each expert group processes the complete hidden layer sequence, and the activated experts in each expert group process the input hidden layer sequence to obtain the output of the activated experts, and based on the output of each activated expert, the output of the hybrid expert layer is obtained, and the output is passed back until the last layer obtains the output of the network, and the loss function is calculated based on this for training. Obviously, the solution of the present application naturally ensures the absolute balance of calculations between devices by grouping experts and evenly distributing them to different devices, and each expert group processes the complete hidden layer sequence, thereby avoiding the calculation waiting between different devices caused by uneven load, and maximizing the utilization of hardware devices. And the effect of the model will not be affected by drop-pad (i.e. discarding data and filling 0).

[0065] Assuming G is the number of expert groups after division and k is the number of activated experts, the number of experts in each group is 1 / G of the original number, and the number of activated experts in the group is reduced to k / G, thereby reducing the memory consumption of token sorting and distribution in each group, so that each group of experts can process the complete sequence, thereby making the load between groups absolutely balanced.

[0066] This embodiment describes the process of obtaining the output of the mixed expert layer based on the output of each activated expert in each expert group in the above step S130.

[0067] In one possible implementation, each expert group can be subjected to normalized Softmax processing according to the scores of each activated expert in the group, and the outputs of each activated expert can be weightedly fused using the normalized weighted probability to obtain the output of each expert group. Finally, the outputs of each expert group are fused to obtain the output of the mixed expert layer.

[0068] In another possible implementation, in order to reduce information redundancy between groups and ensure sufficient diversity among experts, cross-group Softmax processing can also be performed in this embodiment. Specifically:

[0069] The scores of all activated experts in each expert group are normalized by Softmax processing, and the outputs of each activated expert are weighted fused using the normalized weighted probability. The weighted fusion result is used as the output of the hybrid expert layer.

[0070] In some embodiments of the present application, the process of calculating the score of each expert according to the hidden layer sequence through the routing module in the aforementioned step S110 is described.

[0071] In this embodiment, the hybrid expert network training process can adopt a combination of tensor parallelism and sequence parallelism.

[0072] The input hidden layer sequence x of the hybrid expert layer is divided into several subsequences according to the tensor parallel size: r ∈R S / T，d .

[0073] Among them, d represents the hidden dimension of the hidden layer sequence x, S is the sequence length, r represents the rankid, and T represents the tensor parallel size, which is consistent with the number of devices.

[0074] For example, assuming that the number of devices is 8, the tensor parallel size T=8, and the hidden layer sequence x length S=8192, the length of the sequence divided on each device is S / T=8192 / 8=1024.

[0075] Each divided subsequence is input to a device. On each device, the score of each expert is calculated based on the subsequence input by the current device through the shared routing module, and the subsequences input by other devices are obtained by the all-gather communication method to obtain the complete hidden layer sequence x; the score of each expert calculated by other devices is obtained by the all-gather communication method to obtain the score of each expert of the complete hidden layer sequence x.

[0076] In a possible implementation, considering that the device does not need the complete hidden layer sequence x when calculating the expert score using the routing module, the process of each device obtaining the subsequence input by other devices can adopt an asynchronous all-gather communication method, which will not block the process and improve the training efficiency.

[0077] Each device needs to wait for the expert scores calculated by other devices before executing subsequent processing. Therefore, the process of each device obtaining the scores of each expert calculated by other devices can specifically adopt a synchronous all-gather communication method.

[0078] The traditional method uses an All-to-All communication method in the token distribution process. Affected by load imbalance, the communication volume between experts is usually inconsistent, and there is communication imbalance between different devices. In addition, the efficiency of cross-device All-to-All communication is low. In the method provided in this embodiment, the complete hidden layer sequence x is obtained through the all-gather communication method, which ensures the absolute balance of communication volume between different devices.

[0079] Among them, All-gather is a collective communication operation that aims to collect the data of each process (or node) to all processes. This means that at the end of the All-gather operation, each process will have the data of all other processes.

[0080] All-to-all is a more complex and extensive collective communication operation designed to allow each process to send different chunks of data to all other processes and receive chunks of data from all other processes simultaneously. This means that all processes communicate directly with each other.

[0081] For the MOE model, one of the technical routes is to achieve token distribution through all-to-all communication. For example, there are 4 devices (graphics cards), each with 32 experts. The input hidden layer sequence length of the mixed expert layer is 8192.

[0082] Initially, each graphics card has 8192 / 4=2048 tokens. Then, for any 2048 tokens of a graphics card, they need to be distributed to other graphics cards. That is, graphics card 1 needs to send tokens to graphics cards 2, 3, and 4; similarly, graphics card 1 needs to receive tokens sent by graphics cards 2, 3, and 4. Each graphics card needs to send once to all and receive once from all.

[0083] For sending, each graphics card needs to sort the tokens according to the target. Similarly, each graphics card needs to sort the received tokens again (although the tokens sent by each graphics card have been sorted by experts, the three groups of tokens received are not all sorted together, so the tokens retained by itself and the three groups of tokens received must be globally sorted once) in order to perform efficient matrix multiplication. After the calculation is completed, it is necessary to restore the original order and send it back; similarly, it will receive the tokens it sent out at the beginning. At this time, each graphics card has its own initial 2048 tokens, but they are also sorted by experts, and the original sequence needs to be restored once to get the final result.

[0084] Therefore, compared with the double sorting and re-sorting of all-to-all communication, all-gather has more advantages in the implementation of MOE model algorithm.

[0085] In addition, the MOE model algorithm implemented by all-to-all communication has very strict requirements for load balancing. Uneven token distribution will directly lead to different calculation time for each graphics card and different memory consumption for each graphics card. This is harmful to model training, both in terms of efficiency and model effect.

[0086] Next, combine Figure 3 An optional process of the hybrid expert network training method of the present application is introduced.

[0087] For a transformer encoder block, the hidden layer representation u is calculated through the self-attention module to obtain the input of the mixed expert layer, denoted as x. For each tensor parallel rank, there is x r ∈R S / T，d .

[0088] First, each device inputs the hidden layer sequence xr and calculates the expert score of each path through the shared routing module (Gate g):

[0089] s r =g(x r )∈R S / T，E

[0090] Among them, E is the number of all global experts.

[0091] At the same time, asynchronous all-gather obtains the complete hidden layer sequence representation:

[0092] x=[x 0 ;x 1 ;...;x T-1 ]∈R S，d

[0093] Synchronous all-gather obtains the expert scores of the complete hidden layer sequence:

[0094] s=[s 0 ;s 1 ;...;s T-1 ]∈R S，E

[0095] The expert scores s of the complete hidden layer sequence are grouped according to the expert classification, and the top-k / G activated experts in each expert group are selected:

[0096] s∈RS，G，E / G , idx top_k ∈R S，G，k / G

[0097] Among them, G is the number of groups, k is the total number of activated experts, idx top_k It indicates the number of the top-k activated experts with the highest scores, which can also be expressed as top-kindex.

[0098] For all top-k activated experts in all expert groups, calculate the weighted probability p of each expert:

[0099] p j =Softmax j∈topk(s) (s j )

[0100] Furthermore, the distribution and calculation process of each token contained in the complete hidden sequence x in each expert group is as follows:

[0101] For the top-k index obtained in the previous step, experts in each expert group select and divide the top-k index according to their respective ranks, and sort the token sequence according to the index.

[0102] The sorted tokens are calculated by each activation expert and reordered to restore the original sequence order:

[0103] o j，t =E j，t (x i )

[0104] Among them, t is the subscript of each group, and j is the subscript of the top-k / G experts in the group.

[0105] For the calculation output E of each activation expert obtained in the above steps j，t (x i ), perform weighted fusion according to the weighted probability p of each expert calculated in the previous step to obtain the output y of the hybrid expert layer i as follows:

[0106]

[0107] Since each expert group processes the complete hidden layer sequence x, the calculation of y in the above formula only needs to be accumulated between tensor parallel groups through all-reduce communication.

[0108] In summary, this application proposes the concept of expert grouping for the first time, where multiple devices carry group experts in parallel to achieve strict load balancing between devices, avoid synchronous waiting for communication and computing caused by uneven load, and maximize hardware utilization efficiency.

[0109] The method proposed in this application can ensure that the loads between devices are completely balanced and the communication volume is completely consistent, thereby eliminating the need to constrain the load balancing and communication balancing losses between devices.

[0110] This case uses the all-gather communication method to obtain data between devices, and the communication volume between devices is naturally balanced. The traditional expert parallel structure uses all-to-all communication to achieve token distribution. Its communication library optimization is inferior to the reduce-gather series. The communication between nodes becomes a bottleneck, and it requires that the expert division within the device is almost balanced to achieve higher computing efficiency.

[0111] The hybrid expert network training device provided in an embodiment of the present application is described below. The hybrid expert network training device described below and the hybrid expert network training method described above can be referenced to each other.

[0112] See also Figure 4 , Figure 4 A schematic diagram of the structure of a hybrid expert network training device disclosed in an embodiment of the present application.

[0113] For the hybrid expert layer in the hybrid expert network to be trained, all experts are evenly divided into several groups, and each expert group is evenly distributed to several devices, and different devices share the same routing module. Figure 4 As shown, the device may include:

[0114] The data input unit 11 is used to input the training data into the hybrid expert network to obtain the hidden layer sequence output by the previous layer of the hybrid expert layer as the input of the hybrid expert layer;

[0115] A first calculation unit 12 is used to calculate the score of each expert according to the hidden layer sequence through the routing module, and for each expert group, select the activated experts in the group according to the scores of each expert in the group, and the number of activated experts in different groups is the same;

[0116] The second calculation unit 13 is used to input the hidden layer sequence into each expert group respectively, and the activated experts in the expert group process it to obtain the output of each activated expert in each expert group;

[0117] The third calculation unit 14 is used to obtain the output of the mixed expert layer based on the output of each activated expert in each expert group, and pass it backward until the output of the network is obtained through the last layer, calculate the loss function based on the output of the network and the true value label, and update the network parameters according to the loss function.

[0118] In a possible implementation, the process of obtaining the output of the mixed expert layer by the third computing unit based on the output of each activated expert in each expert group includes:

[0119] The scores of all activated experts in each expert group are normalized, and the outputs of each activated expert are weighted fused using the normalized weighted probability. The weighted fusion result is used as the output of the hybrid expert layer.

[0120] In a possible implementation, the hybrid expert network training process adopts a tensor parallel method, and the first computing unit calculates the score of each expert according to the hidden layer sequence through the routing module, including:

[0121] Dividing the hidden layer sequence into a plurality of subsequences according to the tensor parallel size, and inputting each subsequence into a device respectively, wherein the tensor parallel size is consistent with the number of the devices;

[0122] On each device, the score of each expert is calculated based on the subsequence input by the current device through a shared routing module, and the subsequences input by other devices are obtained by an all-gather communication method to obtain a complete hidden layer sequence. The score of each expert calculated by other devices is obtained by an all-gather communication method to obtain the score of each expert in the complete hidden layer sequence.

[0123] In one possible implementation, the process in which the first computing unit uses an all-gather communication method to obtain a subsequence input by other devices includes: using an asynchronous all-gather communication method to obtain a subsequence input by other devices; the process in which the first computing unit uses an all-gather communication method to obtain a score of each expert calculated by other devices includes: using a synchronous all-gather communication method to obtain a score of each expert calculated by other devices.

[0124] In a possible implementation, the third computing unit updates the network parameters according to the loss function, where the loss function also includes:

[0125] The global expert load balancing loss function is used to constrain the output distribution of the routing module so that the input hidden layer sequence can be evenly distributed to all experts.

[0126] The present application also provides an electronic device in an embodiment. Figure 5As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to a fixed terminal such as a computer, a server, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0127] like Figure 5 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the hybrid expert network training method of the aforementioned embodiment of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in RAM 603. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0128] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0129] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any hybrid expert network training method provided in the embodiment of the present application.

[0130] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any hybrid expert network training method provided in the embodiment of the present application.

[0131] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0132] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0133] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0134] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0135] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

Claims

1. A hybrid expert network training method, characterized in that: For a hybrid expert layer in a hybrid expert network to be trained, all experts therein are evenly divided into a plurality of groups, and each of the divided expert groups is evenly distributed to a plurality of devices, and different devices share the same routing module, and the method includes: Sending training data to the hybrid expert network to obtain a hidden layer sequence output by the previous layer of the hybrid expert layer as input to the hybrid expert layer; The routing module calculates the score of each expert according to the hidden layer sequence, and for each expert group, selects activated experts in the group according to the scores of each expert in the group, and the number of activated experts in different groups is the same; Inputting the hidden layer sequence into each of the expert groups respectively, and processing it by the activated experts in the expert group, so as to obtain the output of each activated expert in each of the expert groups; Based on the output of each activated expert in each expert group, the output of the mixed expert layer is obtained, and is passed backward until the output of the network is obtained through the last layer. The loss function is calculated based on the output of the network and the true value label, and the network parameters are updated according to the loss function.

2. The method according to claim 1, characterized in that The process of obtaining the output of the mixed expert layer based on the output of each activated expert in each expert group includes: The scores of all activated experts in each expert group are normalized, and the outputs of each activated expert are weightedly fused using the normalized weighted probability, and the weighted fusion result is used as the output of the hybrid expert layer.

3. The method according to claim 1, characterized in that The hybrid expert network training process adopts a tensor parallel method, and the process of calculating the score of each expert according to the hidden layer sequence through the routing module includes: Dividing the hidden layer sequence into a plurality of subsequences according to the tensor parallel size, and inputting each subsequence into a device respectively, wherein the tensor parallel size is consistent with the number of the devices; On each device, the score of each expert is calculated based on the subsequence input by the current device through the shared routing module, and the subsequences input by other devices are obtained by an all-gather communication method to obtain the complete hidden layer sequence. The score of each expert calculated by other devices is obtained by an all-gather communication method to obtain the score of each expert in the complete hidden layer sequence.

4. The method according to claim 3, characterized in that The process of using the all-gather communication mode to obtain the subsequence input by other devices includes: Use asynchronous all-gather communication to obtain subsequences input from other devices; The process of using the all-gather communication mode to obtain the scores of each expert calculated by other devices includes: A synchronous all-gather communication method is used to obtain the scores of each expert calculated by other devices.

5. The method according to claim 1, characterized in that The loss function of the hybrid expert network training process also includes: The global expert load balancing loss function is used to constrain the output distribution of the routing module so that the input hidden layer sequence can be evenly distributed to all experts.

6. A hybrid expert network training device, characterized in that: For a hybrid expert layer in a hybrid expert network to be trained, all experts therein are evenly divided into several groups, and each of the divided expert groups is evenly distributed to several devices, and different devices share the same routing module. The device includes: A data input unit, used for sending training data into the hybrid expert network to obtain a hidden layer sequence output by a previous layer of the hybrid expert layer as input of the hybrid expert layer; A first calculation unit is used to calculate the score of each expert according to the hidden layer sequence through the routing module, and for each expert group, select an activated expert in the group according to the score of each expert in the group, and the number of activated experts in different groups is the same; A second calculation unit is used to input the hidden layer sequence into each of the expert groups respectively, and the activated experts in the expert groups process it to obtain the output of each activated expert in each of the expert groups; The third computing unit is used to obtain the output of the mixed expert layer based on the output of each activated expert in each expert group, and pass it backward until the output of the network is obtained through the last layer, calculate the loss function based on the output of the network and the true value label, and update the network parameters according to the loss function.

7. The device according to claim 6, characterized in that The process of obtaining the output of the mixed expert layer based on the output of each activated expert in each expert group by the third computing unit includes: The scores of all activated experts in each expert group are normalized, and the outputs of each activated expert are weightedly fused using the normalized weighted probability, and the weighted fusion result is used as the output of the hybrid expert layer.

8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the hybrid expert network training method as described in any one of claims 1 to 5.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the hybrid expert network training method as claimed in any one of claims 1 to 5 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the hybrid expert network training method as described in any one of claims 1 to 5 is implemented.

Images