Expert-parallel processing method and system, electronic device, and storage medium

Abstract

The present application discloses an expert-parallel processing method and system, an electronic device, and a storage medium, which solve the problem of load imbalance on computing cards that carry expert execution, thereby improving the processing efficiency of the computing cards. In the expert-parallel processing system, W experts are deployed from a host node to N computing cards by means of expert replication, wherein each of the N computing cards comprises at least two experts having different expert weights. In an embodiment of the present application, on the basis of text processing units respectively activated by the W experts, M text processing units are assigned to at least two experts on each of the N computing cards, so as to obtain text processing units respectively activated by the at least two experts on each of the N computing cards. The total number of text processing units activated by the at least two experts and corresponding to a same expert weight across different computing cards is equal to the number of text processing units corresponding to the same expert weight among the W experts obtained by routing partitioning.

Description

An expert parallel processing method and system, electronic equipment and storage medium

[0001] The present application claims priority from the Chinese patent application No. 2024113689459 filed on September 27, 2024, and entitled "An expert parallel processing method and system, electronic equipment and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computers, in particular to an expert parallel processing method and system, electronic equipment and storage medium. BACKGROUND

[0003] With the rapid development of large language model (LLM) technology and the complex and diverse application scene requirements, efficient training and inference of large language models are increasingly important. In order to improve model performance and reduce inference cost, for example, a mixture of experts (MoE) structure is used in large language models.

[0004] In a large language model, a gating network is used to calculate which expert to assign to a text processing unit, and a full connection feed-forward network (FFN) is used to calculate using the expert and the text processing unit.

[0005] However, the current large language model is not balanced in assigning experts to text processing units during training and inference, resulting in an imbalance in the load of the computing cards running the experts, which reduces the processing efficiency of the computing cards. SUMMARY

[0006] The present application provides an expert parallel processing method and system, electronic equipment and storage medium, which is used to solve the problem of load imbalance of computing cards running experts and improve the processing efficiency of computing cards.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] In a first aspect, the present application provides an expert parallel processing method, which is applied to an expert parallel processing system, the expert parallel processing system comprising a host node and N computing cards, W experts are stored in the host node, the W experts are deployed from the host node to the N computing cards by expert replication, each computing card in the N computing cards comprises at least two experts with different expert weights, and W and N are positive integers.

[0009] The method comprises: obtaining M text processing units in a large language model, M being a positive integer; performing routing division on the M text processing units to obtain W text processing units activated by W experts respectively; and distributing the M text processing units to at least two experts on each of N computing cards according to the W text processing units activated by the W experts respectively, to obtain text processing units activated by at least two experts on each of the N computing cards respectively, wherein the total number of the text processing units activated by the at least two experts corresponding to the same expert weight on different computing cards is equal to the number of the text processing units corresponding to the same expert weight in the W experts obtained by the routing division.

[0010] In the above scheme, M text processing units are obtained, the M text processing units are subjected to routing division, initial text processing unit distribution of W experts is realized, and then re-distribution of text processing units is performed according to all experts obtained by expert replication of N computing cards. Taking at least two experts on each of the N computing cards as an example, the text processing units corresponding to one expert weight are distributed to the N computing cards by routing division, and since there are replicated experts on the N computing cards, the text processing units corresponding to the same expert weight are distributed on at least two computing cards, so that the text processing units corresponding to the same expert weight can be inferred by the at least two computing cards, thereby solving the problem of load imbalance caused by the fact that one expert can only run on one computing card.

[0011] In a possible implementation manner of the first aspect, the method further comprises:

[0012] Infer using the at least two experts on each of the N computing cards and the text processing units activated by the at least two experts on each of the N computing cards respectively to obtain an output result of the large language model.

[0013] In the above scheme, the at least two experts on each of the N computing cards and the text processing units activated by the at least two experts on each of the N computing cards are used to infer the large language model to obtain an inference result of the large language model. For example, the inference using the at least two experts on each of the N computing cards and the text processing units activated by the at least two experts on each of the N computing cards can be full connection feedforward neural network (FFN) calculation. In the embodiment of the present application, the W experts are deployed to the N computing cards by expert replication, so that the text processing units corresponding to the same expert weight on at least two computing cards are inferred, thereby solving the problem of load imbalance caused by the fact that one expert can only run on one computing card.

[0014] In a possible implementation manner of the first aspect, the W experts are deployed from the host node to the N computing cards in an expert replication manner, including:

[0015] obtaining the W experts stored on the memory in the host node;

[0016] deploying the W experts from the memory of the host node to the N video memories of the N computing cards in an expert replication manner.

[0017] In the foregoing solution, the host node includes the memory, and the computing card includes the video memory. In the embodiment of the application, the original W experts are stored on the memory of the host node. When the W experts are deployed to the N video memories of the N computing cards, the expert replication manner can be used, so that the original W experts and the copied expert copies can be stored on the N computing cards. For example, the expert is replicated X times, and (X+1)M experts are obtained in total. The (X+1)M experts are stored in the N video memories of the N computing cards. In the embodiment of the application, the expert and the corresponding expert copy are used to perform inference on the text processing unit, so as to improve the processing efficiency of the computing card.

[0018] In a possible implementation manner of the first aspect, the M text processing units are allocated to at least two experts on each of the N computing cards according to the text processing units activated by the W experts respectively, to obtain text processing units activated by at least two experts on each of the N computing cards respectively, including:

[0019] obtaining the average number of text processing units activated by an expert according to the text processing units activated by the W experts respectively;

[0020] allocating the number of text processing units corresponding to the same expert weight in the W experts obtained by the routing division to at least two experts corresponding to the same expert weight on different computing cards according to the average number of text processing units activated by the expert.

[0021] In the foregoing solution, after the average number of text processing units activated by an expert is obtained, the number of text processing units corresponding to the same expert weight in the W experts obtained by the routing division is re-allocated according to the average number of text processing units activated by the expert, so that the number of text processing units corresponding to the same expert weight in the W experts is allocated to at least two experts corresponding to the same expert weight on different computing cards. For example, the number of text processing units corresponding to the same expert weight obtained by the routing division can be equally divided according to the average number of text processing units activated by an expert, to be allocated to at least two experts corresponding to the same expert weight on different computing cards.

[0022] In a possible implementation manner of the first aspect, the method further includes: distributing the text processing units corresponding to the same expert weight of the same one of the W experts to at least two experts on different computing cards according to the same expert weight, including:

[0023] determining first text processing units corresponding to a first expert from the text processing units activated by the W experts respectively, the first expert being an expert whose number of activated text processing units is greater than the average number of activated text processing units of the experts, the first expert running on a first computing card of the N computing cards;

[0024] distributing first T text processing units of the first text processing units to the first expert on the first computing card, and distributing other text processing units of the first text processing units except the T text processing units to a second expert on a second computing card of the N computing cards, the first expert and the second expert corresponding to the same expert weight, T representing the average number of activated text processing units of the experts.

[0025] In the above scheme, the first T text processing units are taken out from the first text processing units according to the average number of activated text processing units of the experts, the first T text processing units are distributed to the first expert on the first computing card, and other text processing units of the first text processing units except the T text processing units are distributed to the second expert on the second computing card of the N computing cards, for example, the number of the first text processing units is 16, and T is equal to 10, then 10 text processing units of the first text processing units can be distributed to the first expert, and the remaining 6 text processing units are distributed to the second expert, since the first expert and the second expert correspond to the same expert weight, the first expert and the second expert can jointly use the first text processing units for inference, thereby improving the processing efficiency of the computing card.

[0026] In a possible implementation manner of the first aspect, the text processing units activated by the W experts respectively include W text processing units activated by the W experts in descending order of the number of text processing units activated by the experts.

[0027] In the above scheme, after the M text processing units are routed and divided, the W text processing units activated by the W experts respectively are obtained in descending order of the number of text processing units activated by the experts, and the W text processing units activated by the W experts are obtained in descending order of the number of text processing units activated by the experts in the embodiment of the application, so that the text processing units of each expert are redistributed in turn according to the number of text processing units activated by the experts, thereby improving the efficiency of allocating text processing units to experts.

[0028] In a possible implementation manner of the first aspect, the W experts are deployed from the host node to the N computing cards in an expert replication manner, including:

[0029] obtaining an i-th layer in the large language model, where i is a positive integer;

[0030] deploying the W experts corresponding to the i-th layer from the host node to the N computing cards in an expert replication manner, wherein W experts corresponding to an i+j-th layer in the large language model are stored in the host node, and j is a positive integer.

[0031] In the above solution, in the embodiments of the present application, through expert replication of the W experts corresponding to the i-th layer, more than W experts can be deployed on the N computing cards, for example, each expert is replicated once, and 2W experts can be deployed on the N computing cards, which can be used for inference according to the M text processing units, solving the problem of uneven load of the computing cards, thereby improving the processing efficiency of the computing cards.

[0032] In a possible implementation manner of the first aspect, the M text processing units are distributed to at least two experts on each of the N computing cards according to the text processing units activated by the W experts, respectively, including:

[0033] The M text processing units are distributed to at least two experts on each of the N computing cards according to the text processing units activated by the W experts corresponding to the i-th layer, respectively;

[0034] The method further includes:

[0035] After the M text processing units are distributed to at least two experts on each of the N computing cards, in the process of using at least two experts on each of the N computing cards for inference, the W experts corresponding to the i+j-th layer are deployed from the host node to the N computing cards in an expert replication manner.

[0036] In the above scheme, in the process of reasoning by at least two experts on each of the N computing cards corresponding to the i-th layer, the W experts corresponding to the i+j-th layer are deployed from the host node to the N computing cards in the form of expert replication. The large language model includes multiple layers. When calculating the i-th layer, the W experts corresponding to the i+j-th layer can be deployed from the host node to the N computing cards in the form of expert replication, so that the reading time can be covered by calculation. The large language model has many layers, and a certain time is needed to calculate the result of each layer. Reading the data of the next layer while calculating a layer means that calculation can be performed while reading, so as to achieve the purpose of covering the reading time by calculation and realize the effect of expert parallelism.

[0037] In a possible implementation manner of the first aspect, the method further includes:

[0038] Releasing the W experts of the M text processing units in the i-th layer of the large language model that have been calculated.

[0039] In the above scheme, after reasoning by at least two experts on each of the N computing cards corresponding to the i-th layer, the W experts of the M text processing units in the i-th layer that have been calculated can be released, and the W experts of the M text processing units in the i-th layer that have been calculated are released from the N video memories of the N computing cards. In the embodiment of the application, the experts of the text processing units that have been calculated are released, thereby reducing the memory occupation of the computing card.

[0040] In a second aspect, the embodiments of the present application further provide a processing system for expert parallelism, which includes a host node and N computing cards. The W experts are stored in the host node, and the W experts are deployed from the host node to the N computing cards in the form of expert replication. Each of the N computing cards includes at least two experts with different weights. The W and the N are positive integers.

[0041] The host node includes:

[0042] The acquisition module is configured to acquire M text processing units token in a large language model, and the M is a positive integer.

[0043] The routing division module is configured to divide the M text processing units by routing to obtain text processing units activated by the W experts respectively.

[0044] The distribution module is configured to distribute the M text processing units to at least two experts on each of the N computing cards according to the text processing units activated by the W experts respectively, so as to obtain text processing units activated by at least two experts on each of the N computing cards respectively, and the total number of the text processing units activated by the at least two experts corresponding to the same expert weight on different computing cards is equal to the number of the text processing units corresponding to the same expert weight in the W experts obtained by the routing division.

[0045] In the above scheme, the M text processing units are obtained, the routing division is performed on the M text processing units, the initial text processing unit distribution of the W experts is implemented, and then the re-distribution of the text processing units is performed according to all the experts obtained by the expert replication of the N computing cards. Taking the case that each of the N computing cards includes at least two experts as an example, the routing division is performed to distribute the text processing units corresponding to one expert weight to the N computing cards. Since the experts are replicated on the N computing cards, the experts corresponding to the same expert weight are distributed on at least two computing cards, and thus the text processing units corresponding to the same expert weight can be inferred by the at least two computing cards, thereby solving the problem of unbalanced load caused by the fact that one expert can only run on one computing card.

[0046] In a second aspect of the present application, the component module of the expert parallel processing system can also perform the steps described in the foregoing first aspect and various possible implementation manners, and details are described in the foregoing description of the first aspect and various possible implementation manners.

[0047] In a third aspect, the embodiments of the present application provide a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are run on a computer, the computer is caused to execute the method in the foregoing first aspect.

[0048] In a fourth aspect, the embodiments of the present application provide a computer program product containing instructions, and when the instructions are run on a computer, the computer is caused to execute the method in the foregoing first aspect.

[0049] In a fifth aspect, the embodiments of the present application provide a communication apparatus, which can include a terminal device or a chip, and the like. The communication apparatus includes a processor, a memory, the memory is configured to store instructions, and the processor is configured to execute the instructions in the memory, so that the communication apparatus executes the method in any one of the foregoing first aspect.

[0050] In a sixth aspect, the present application provides a chip system, which comprises a processor for supporting the expert parallel processing system to implement the functions involved in the above aspects, such as sending or processing the data and / or information involved in the above method. In a possible design, the chip system further comprises a memory, and the memory is configured to store the program instructions and data necessary for the expert parallel processing system. The chip system can be composed of a chip, or can comprise a chip and other discrete devices.

[0051] In a seventh aspect, the embodiments of the present application provide a chip, which comprises one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of an electronic device and send a signal to the processor, and the signal comprises computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device is caused to perform the method in the first aspect or any possible implementation manner of the first aspect.

[0052] The seventh aspect and any possible implementation manner of the seventh aspect correspond to the first aspect and any possible implementation manner of the first aspect respectively. For details, refer to the technical effects of the first aspect and any possible implementation manner of the expert parallel processing method of the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0053] FIG. 1 is a schematic diagram of an architecture of an expert parallel processing system according to an embodiment of the present application;

[0054] FIG. 2 is a schematic diagram of a flow of an expert parallel processing method according to an embodiment of the present application;

[0055] FIG. 3 is a schematic diagram of an execution flow of an expert parallel processing system according to an embodiment of the present application;

[0056] FIG. 4 is a schematic diagram of deploying three experts to four cards in a manner of expert replication according to an embodiment of the present application;

[0057] FIG. 5 is a schematic diagram of an architecture of a MoE network according to an embodiment of the present application;

[0058] FIG. 6 is a schematic diagram of deploying experts on each layer in a MoE network according to an embodiment of the present application;

[0059] FIG. 7 is a schematic diagram of deploying experts corresponding to each layer to four cards in a MoE network according to an embodiment of the present application;

[0060] FIG. 8 is a schematic diagram of a structure of an expert parallel processing system according to an embodiment of the present application;

[0061] FIG. 9 is a structural schematic diagram of a host node according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] The embodiments of the present application provide a processing method and system for experts in parallel, an electronic device and a storage medium, which are used for solving the problem of load imbalance of the computing card running the expert, and improving the processing efficiency of the computing card.

[0063] The embodiments of the present application are described below in conjunction with the accompanying drawings. The terms used in the implementation part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0064] Those of ordinary skill in the art can know that, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0065] In the present application, "at least one" means one or more, and "multiple" means two or more. The terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a way of distinguishing the objects with the same attributes used in the description of the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units not clearly listed or inherent to these processes, methods, products or devices.

[0066] A large language model (LLM) is an artificial intelligence model that uses a large amount of text data to train to generate natural language text or understand the meaning of language text. Large language models have shown high efficiency in many tasks and have become an important way to artificial intelligence. Large language models mainly include the following types: pre-trained language models (Pretrained Language Model), neural network language models (Neural Network Language Model), deep learning language models (Deep Learning Language Model) and hybrid language models (Hybrid Language Model).

[0067] The expert parallel processing method provided in the embodiments of the present application is applicable to a large language model. In order to improve the model effect and reduce the inference cost, a mixture of experts (MoE) structure can be used in the large language model. The MoE structure can greatly expand the model capacity with minimum computational overhead. The large language model in the embodiments of the present application adopts the MoE structure only as a feasible implementation manner, and does not limit the embodiments of the present application.

[0068] The mixture of experts (MoE) architecture is a neural network model architecture that improves efficiency and performance by dynamically activating a subset of specialized networks called "experts" for each input. A gating network determines which experts to activate, resulting in sparse activation and reduced computational cost. The MoE architecture consists of two key components: a gating network and expert networks. Each task in the MoE architecture is routed to the most suitable expert or sub-model that specializes in handling that particular task. This dynamic routing ensures that the most capable resources are used for each task, thereby improving the overall efficiency and effectiveness of the model. The MoE architecture completes tasks through multiple experts, and the MoE increases the parameter size of the model by adding more parameters to each expert. The MoE changes the traditional neural network architecture, containing a gating network to determine which experts are used for a specified task. Each artificial intelligence model is fine-tuned to a certain extent, so each expert in the MoE is fine-tuned to achieve the expected effect of the additional adjustment layer that traditional models cannot utilize.

[0069] The network model adopting the MoE structure has the following advantages compared to the dense network model, such as faster pre-training speed; the network model adopting the MoE structure has faster inference speed under the same parameter amount, and only activates part of the experts during inference, such as only activating the top 2 experts. In addition, the network model adopting the MoE structure can easily increase the overall network parameters, and the network size can be expanded to trillion parameters. Finally, the network model adopting the MoE structure has good multi-task learning ability.

[0070] The network model adopting the MoE structure includes a sparse MoE layer and a gating network, wherein the sparse MoE layer replaces the full connection feed-forward neural network (FFN) layer in the traditional Transformer model, and a group of experts are included in the FFN layer, for example, the number of experts is 8. The gating network is used to calculate which expert the token should be sent to.

[0071] The network model adopting the MoE structure has a memory consumption problem. The network model adopting the MoE structure occupies an order of magnitude higher memory than the dense network model. For example, the memory consumption of the Switch Transformer model is 75 times that of the dense T5 model, which causes the network model adopting the MoE structure to be unable to complete single computing card deployment and requires the network model adopting the MoE structure to be deployed on multiple computing cards.

[0072] The network model adopting the MoE structure has a load imbalance problem. When experts are selected, some experts are more likely to be selected by tokens. Therefore, popular experts need to process more tokens, which causes some experts to process more tokens and take longer to calculate, and some experts to process fewer tokens and calculate slowly. The distribution of expert popularity is uneven, with a maximum difference of more than 10 times. The overloaded experts become the tailing bottleneck, and therefore, the expert load of the MoE structure needs to be optimized to improve the MoE parallel inference performance.

[0073] Based on the above analysis, the current MoE structure has an uneven load imbalance problem in the training and inference processes, which reduces the efficiency of computer processing.

[0074] In the current network model adopting auxiliary loss, an auxiliary loss function is added in the training process of the model to guide the importance of each expert and the number of tokens received in the model training, thereby avoiding overuse of experts. Since the auxiliary loss function is used to balance the allocation of experts, when the load is unbalanced during model training, the loss value will increase, which will bring a certain penalty to the network model, thereby avoiding overuse of some experts.

[0075] The method of adjusting expert allocation by using auxiliary loss value is only applicable to the training process of the model and cannot be applied to the inference of the model. Because the auxiliary loss value is used to update the parameters of the model, the auxiliary loss value can be used to measure the difference between the predicted results and the true results of the model in the model training process, and the auxiliary loss value can be used to adjust the target of the model. However, in the inference process of the model, the parameters of the model are not updated, and there is no process of calculating the loss function. Therefore, the auxiliary loss value cannot be used to adjust the load balance in the inference process of the model.

[0076] Embodiments of the present application provide a kind of expert parallel processing method, the method is applied to expert parallel processing system, expert parallel processing system can adopt Host-Device mode, as shown in Figure 1, a kind of expert parallel processing system provided by the present application includes: host node and N computing card, the type of host node is not limited here, exemplarily, host node can be the computing device including memory, for example, the computing device can be terminal device, can also be cloud server etc..Computing card refers to the computing card including video memory, for example, a computing card is single shengteng computing card used in model inference process, computing card in the present application embodiment can also be other types of computing card, not limited here.

[0077] As shown in Figure 1, the present application provides a kind of expert parallel processing system, including: host node and N computing card, W experts are stored in host node, expert refers to the sub-model capable of carrying out model inference processing a kind of problem according to token (token).W is used to represent the number of experts stored in host node in the present application embodiment, the value of W is not limited in the present application embodiment.For example, when it involves artificial intelligence, each expert in large language model is specially studied one more macroscopic problem, and the expert for solving specific problem improves inference efficiency, and enhances the effectiveness and accuracy of network model.

[0078] N is used to represent the number of computing cards in the present application embodiment, the value of N is not limited in the present application embodiment.Each computing card can run one or more experts in the present application embodiment.

[0079] W experts are deployed from host node to N computing cards by expert replication in the present application embodiment, at least two experts with different expert weights are included on each computing card in N computing cards, W and N are positive integers.Expert replication refers to the replication based on an expert weight expert to obtain an expert weight expert copy, and expert replication can also be called expert copy.Only one expert is allocated to one computing card in the present application embodiment, and the same expert weight expert can be replicated to multiple computing cards by expert replication.For example, one expert weight expert and expert copy can be generated by replicating an expert once, the expert can be allocated to a computing card, and the expert copy can be allocated to another computing card, so at least two experts with different expert weights are included on each computing card in N computing cards in the present application embodiment.

[0080] It can be understood that the expert copy defined above is also an expert, for example, for an expert and an expert copy with the same expert weight, both refer to experts with the same expert weight, and experts and expert copies can be collectively referred to as experts in subsequent embodiments of the present application, for example, expert 0 and expert copy 0 can both refer to experts with an expert weight of 0.

[0081] It can be understood that, in the embodiments of the present application, the expert copy can be one expert copy or multiple expert copies. For example, one copy of the expert 0 can be made to one expert copy 0, and X copies of the expert 0 can be made to X expert copies. In the embodiments of the present application, X copies of the expert can be made to obtain X+1 experts, and the value of X is not limited.

[0082] As shown in FIG. 1, taking one copy of the same expert as an example, W experts are stored in the host node, which are expert 1, expert 2, …, and expert W. One copy of the same expert can obtain two experts with the same weight. One copy of the expert 1 can obtain two experts 1, one copy of the expert 2 can obtain two experts 2, and one copy of the expert W can obtain two experts W. Through copying, the expert 1 and the expert 2 can be allocated to the computing card 0, the expert 2 and the expert 3 can be allocated to the computing card 1, and so on, the expert W-1 and the expert W can be allocated to the computing card N-1, and the expert W and the expert 1 can be allocated to the computing card N.

[0083] In some embodiments of the present application, the W experts are deployed from the host node to the N computing cards by using the expert copy, including:

[0084] Obtaining W experts stored on the memory of the host node;

[0085] Deploying the W experts from the memory of the host node to the N video memories of the N computing cards by using the expert copy.

[0086] The host node includes the memory, and the computing card includes the video memory. In the embodiments of the present application, the original W experts are stored on the memory of the host node, and the W experts are deployed to the N video memories of the N computing cards by using the expert copy. Thus, the original W experts and the copied expert copies can be stored on the N computing cards. For example, X copies of the expert are made, and (X+1)M experts are obtained. The (X+1)M experts are stored in the N video memories of the N computing cards. In the embodiments of the present application, the expert and the corresponding expert copy can be used to infer the text processing unit, thereby improving the processing efficiency of the computing card.

[0087] As shown in FIG. 2, the embodiment of the present application provides a processing method of expert parallelism, the processing method of expert parallelism is applied to a processing system of expert parallelism, the processing system of expert parallelism includes: a host node and N computing cards, W experts are stored in the host node, the W experts are deployed from the host node to the N computing cards by means of expert replication, each computing card in the N computing cards includes at least two experts with different expert weights, and W and N are positive integers.

[0088] The processing method of expert parallelism provided by the embodiment of the present application includes the following steps:

[0089] 201. M text processing units in a large language model are obtained, and M is a positive integer.

[0090] In the large language model, multiple text processing units can be input. Taking the number of text processing units as M for example, the value of M is not limited in the embodiment of the present application. In natural language processing, a token refers to the smallest semantic unit in a text. The token can be referred to as a text processing unit. For example, a sentence can be divided into several words, and each word is a token.

[0091] The way of obtaining M text processing units is not limited in the embodiment of the present application. For example, M text processing units can be obtained by an All-Reduce operation.

[0092] 202. The M text processing units are routed and divided to obtain W text processing units activated by W experts respectively.

[0093] In the large language model, a gating network is provided. The M text processing units are routed and divided to obtain W text processing units activated by W experts respectively. The number of text processing units activated by W experts respectively needs to be determined in combination with a specific application scenario, and the number of text processing units activated by W experts respectively is not limited here.

[0094] For example, taking W=4 for example, four experts are expert 0, expert 1, expert 2 and expert 3. Expert 0 activates 8 text processing units, expert 1 activates 5 text processing units, expert 2 activates 16 text processing units, and expert 3 activates 11 text processing units.

[0095] For example, taking W=4 for example, four experts are expert 0, expert 1, expert 2 and expert 3. Expert 0 activates 0 text processing units, expert 1 activates 0 text processing units, expert 2 activates 0 text processing units, and expert 3 activates 40 text processing units.

[0096] 203. Based on the text processing units activated by W experts, assign M text processing units to at least two experts on each of the N computing cards, so as to obtain text processing units activated by at least two experts on each of the N computing cards.

[0097] The total number of text processing units activated by at least two experts corresponding to the same expert weight on different computing cards is equal to the number of text processing units corresponding to the same expert weight among the W experts obtained by routing.

[0098] In this embodiment, step 202 allows for the routing of M text processing units, achieving the initial allocation of text processing units to W experts. Next, the text processing units need to be redistributed based on all experts obtained through expert replication across N computing cards. Taking an example where each of the N computing cards contains at least two experts, the text processing unit corresponding to an expert weight obtained through routing in step 202 is allocated across the N computing cards. Since replicated experts exist on the N computing cards, experts with the same weight are distributed across at least two computing cards. Therefore, inference can be performed on the text processing unit corresponding to the same expert weight using these at least two computing cards, solving the load imbalance problem caused by an expert only running on one computing card.

[0099] For example, consider the scenario where expert 1 is copied once to create two experts 1. Among N computing cards, computing card 1 includes experts 0 and 1, and computing card 2 includes experts 1 and 2. During routing, expert 1 is allocated 40 text processing units. If expert 1 is assigned to one computing card using the current method, that card would need to process 40 text processing units. However, according to the expert copying method in this embodiment, expert 1 on computing card 1 is allocated 20 text processing units, and expert 1 on computing card 2 is allocated 20 text processing units. Therefore, this embodiment can solve the problem of unbalanced load on computing cards and improve the processing efficiency of the computing cards.

[0100] In some embodiments of this application, in addition to performing the aforementioned steps 201 to 203, the method performed by the expert parallel processing system may also include the following steps:

[0101] A1. Use at least two experts on each of the N computing cards and the text processing units activated by at least two experts on each computing card to perform inference, so as to obtain the output of the large language model.

[0102] Wherein, after obtaining the at least two text processing units respectively activated by the at least two experts on each of the N computing cards, inference of the large language model is performed using the at least two experts on each of the N computing cards and the at least two text processing units respectively activated by the at least two experts on each of the N computing cards to obtain an inference result of the large language model. For example, the inference using the at least two experts on each of the N computing cards and the at least two text processing units respectively activated by the at least two experts on each of the N computing cards can be performing full connection feedforward neural network (FFN) calculation. In the embodiments of the present application, the W experts are deployed on the N computing cards in the manner of expert replication, so that the text processing units corresponding to the same expert weight on at least two computing cards perform inference, thereby solving the problem of unbalanced load caused by the fact that one expert can only run on one computing card.

[0103] Further, in some embodiments of the present application, step 203 distributes the M text processing units to the at least two experts corresponding to the same expert weight on different computing cards according to the W experts respectively activated text processing units, to obtain the at least two text processing units respectively activated by the at least two experts on each of the N computing cards, comprising the following steps B1 and B2:

[0104] B1, obtaining the average number of text processing units activated by experts according to the W experts respectively activated text processing units.

[0105] Wherein, the average number of text processing units activated by experts refers to the average value of the number of text processing units respectively activated by the W experts, and the average number of text processing units activated by experts can measure the processing load of the W experts processing the M text processing units.

[0106] For example, W is equal to 4, and the host node stores 4 experts, which are expert 0, expert 1, expert 2 and expert 3. When performing routing division, it is determined that expert 0 is divided into 8 tokens, expert 1 is divided into 5 tokens, expert 2 is divided into 16 tokens, and expert 3 is divided into 11 tokens, so a total of 8+5+16+11=40 tokens are obtained. The 40 tokens and the 4 experts are evenly divided, and on average, each computing card needs to calculate 10 tokens, that is, the average number of text processing units activated by experts is equal to 10 tokens.

[0107] B2, distributing the number of text processing units corresponding to the same expert weight in the W experts obtained by routing division to the at least two experts corresponding to the same expert weight on different computing cards.

[0108] wherein, after obtaining the average number of activated text processing units of the experts, the number of text processing units corresponding to the same expert weight in the W experts obtained by the routing is redistributed according to the average number of activated text processing units of the experts, so that the text processing units corresponding to the same expert weight in the W experts are distributed to at least two experts corresponding to the same expert weight on different computing cards. For example, referring to the average number of activated text processing units of the experts, the text processing units corresponding to the same expert weight obtained by the routing can be evenly divided and distributed to at least two experts corresponding to the same expert weight on different computing cards.

[0109] Further, in some embodiments of the present application, step B2 distributes the number of text processing units corresponding to the same expert weight in the W experts obtained by the routing to at least two experts corresponding to the same expert weight on different computing cards according to the average number of activated text processing units of the experts, comprising:

[0110] B21, determining the first text processing unit corresponding to the first expert from the text processing units activated by the W experts, the first expert being an expert whose number of activated text processing units is greater than the average number of activated text processing units of the experts, the first expert running on a first computing card of the N computing cards.

[0111] wherein, according to the average number of activated text processing units of the experts, the first expert whose number of activated text processing units is greater than the average number of activated text processing units of the experts is determined from the text processing units activated by the W experts, the first text processing unit being obtained by the routing, the number of the first text processing unit being greater than the average number of activated text processing units of the experts, the first expert running on a first computing card of the N computing cards, based on the foregoing description of expert replication, there are at least two experts with the same expert weight, for example, the same expert weight corresponds to the first expert and the second expert, the first expert running on the first computing card, and the second expert running on a second computing card of the N computing cards, the second expert being considered as a copy of the first expert, or the first expert being a copy of the second expert.

[0112] B22, distributing the first T text processing units in the first text processing unit to the first expert on the first computing card, and distributing the other text processing units in the first text processing unit except for the T text processing units to the second expert on a second computing card of the N computing cards, the first expert and the second expert corresponding to the same expert weight, T representing the average number of activated text processing units of the experts.

[0113] Wherein, the number of text processing units activated by an expert is T, T is the average value Avg, the number of text processing units of the first text processing unit is greater than the number of text processing units activated by the expert, and the first T text processing units are taken out from the first text processing unit according to the number of text processing units activated by the expert, and the first T text processing units are allocated to the first expert on the first computing card, and the other text processing units in the first text processing unit are allocated to the second expert on the second computing card in the N computing cards, for example, the number of the first text processing unit is 16, and T is equal to 10, then 10 text processing units in the first text processing unit can be allocated to the first expert, and the remaining 6 text processing units can be allocated to the second expert, since the first expert and the second expert correspond to the same expert weight, the first expert and the second expert can jointly use the first text processing unit for reasoning, thereby improving the processing efficiency of the computing card.

[0114] In the embodiment of the application, the W text processing units activated by the experts respectively include W text processing units activated by the experts respectively in descending order of the number of text processing units activated by the experts.

[0115] Wherein, after the M text processing units are routed and divided, the W text processing units activated by the experts respectively can be sorted in descending order of the number of text processing units activated by the experts, and then the W text processing units activated by the experts respectively are obtained. In the embodiment of the application, the W text processing units activated by the experts respectively are sorted in descending order of the number of text processing units activated by the experts, so that the text processing units of each expert can be redistributed in turn according to the number of text processing units activated by the experts, thereby improving the efficiency of allocating text processing units to experts.

[0116] In some embodiments of the application, the large language model includes a plurality of layers, which can be a calculation layer in the large language model. For example, the large language model can be a network model using MoE structure, and the network model using MoE structure includes MoE layers. Each layer included in the large language model can deploy W experts. Specifically, the W experts are deployed from the host node to the N computing cards in the form of expert replication, including:

[0117] C1, obtaining the i-th layer in the large language model, i is a positive integer;

[0118] C2, deploying W experts corresponding to the i-th layer from the host node to the N computing cards in the form of expert replication, wherein W experts corresponding to the i+j-th layer in the large language model are stored in the host node, and j is a positive integer.

[0119] Among them, the large language model includes multiple layers, for example, the large language model can include the i-th layer and the i+j-th layer, and the values of i and j are not limited, for example, i is equal to 0 and j is equal to 1, then the W experts corresponding to the 0th layer can be deployed from the memory of the host node to the N memory of the N computing cards in the form of expert replication, and the W experts corresponding to the 1st layer are still stored in the memory of the host node, so as to reduce the occupation of the N memory.

[0120] In the embodiments of the application, through expert replication of the W experts corresponding to the i-th layer, more than W experts can be deployed on the N computing cards, for example, each expert is replicated once, and 2W experts can be deployed on the N computing cards, which can be used for inference according to the M text processing units, solving the problem of uneven load of the computing cards, thereby improving the processing efficiency of the computing cards.

[0121] Further, in some embodiments of the application, step 203 distributes the M text processing units to at least two experts on each computing card in the N computing cards according to the text processing units activated by the W experts respectively, comprising:

[0122] D1, distributing the M text processing units to at least two experts on each computing card in the N computing cards according to the text processing units activated by the W experts corresponding to the i-th layer respectively.

[0123] Among them, the M text processing units are routed and divided to realize the initial text processing unit distribution of the W experts corresponding to the i-th layer, and then the text processing unit needs to be distributed again according to all the experts obtained by expert replication of the N computing cards. Taking the example that each computing card in the N computing cards includes at least two experts, the text processing units corresponding to the expert weight of the i-th layer obtained by routing and division are distributed to the N computing cards, and since there are replicated experts on the N computing cards, the experts with the same expert weight are distributed on at least two computing cards, so the at least two computing cards can infer the text processing units corresponding to the same expert weight, solving the problem of uneven load caused by the fact that one expert can only run on one computing card.

[0124] In some embodiments of the application, in addition to performing the foregoing D1, the expert parallel processing system can also perform the following steps in the method performed by the expert parallel processing system:

[0125] E1, after distributing the M text processing units to at least two experts on each computing card in the N computing cards, in the process of using at least two experts on each computing card in the N computing cards for inference, the W experts corresponding to the i+j-th layer are deployed from the host node to the N computing cards in the form of expert replication.

[0126] In the process of reasoning by at least two experts on each of the N computing cards corresponding to the i-th layer, the W experts corresponding to the i+j-th layer are deployed from the host node to the N computing cards in the form of expert replication. The large language model includes multiple layers. When calculating the i-th layer, the W experts corresponding to the i+j-th layer can be deployed from the host node to the N computing cards in the form of expert replication, so that the reading time can be covered by calculation. The large language model has many layers, and it takes a certain time to calculate the result of each layer. Reading the data of the next layer while calculating a layer means that calculation can be performed while reading, so as to achieve the purpose of covering the reading time by calculation and realize the effect of expert parallelism.

[0127] Further, in some embodiments of the present application, in addition to performing D1 and E1, the expert parallel processing system can further perform the following steps:

[0128] F1, release the W experts of the M text processing units in the i-th layer of the large language model that have been calculated.

[0129] After reasoning by at least two experts on each of the N computing cards corresponding to the i-th layer, the W experts of the M text processing units in the i-th layer that have been calculated can be released, and the W experts of the M text processing units in the i-th layer that have been calculated can be released from the N video memories of the N computing cards. In the embodiments of the present application, by releasing the experts of the text processing units that have been calculated, the occupation of the video memory of the computing card is reduced.

[0130] As can be known from the foregoing embodiments, the M text processing units are divided by routing, the W experts are allocated to the initial text processing units, and then the text processing units are allocated again according to all the experts obtained by expert replication of the N computing cards. Taking at least two experts on each of the N computing cards as an example, the text processing units corresponding to a certain expert weight are allocated to the N computing cards by routing division. Since there are replicated experts on the N computing cards, the experts of the same expert weight are distributed on at least two computing cards, so that the at least two computing cards can reason the text processing units corresponding to the same expert weight, thereby solving the problem of unbalanced load caused by the fact that one expert can only run on one computing card.

[0131] In order to better understand and implement the above-mentioned schemes of the embodiments of the present application, the following examples of corresponding application scenarios are used for specific description.

[0132] Next, take the large language model as the MoE model, where the MoE model includes multiple MoE layers as an example for illustration. In the following process, the computing card will be simply referred to as the card, and the text processing unit will be expressed as a token.

[0133] When expert parallelism is implemented in the MoE network, the number of tokens assigned to each expert may be inconsistent, resulting in load imbalance. Load imbalance will lead to an increase in inference latency and a waste of hardware resources.

[0134] In the embodiment of this application, the expert copy deployment method is adopted. According to the hardware topology, the experts are replicated and distributed on multiple cards. In the embodiment of this application, the routing partitioning algorithm is adopted. The experts are sorted in descending order according to the number of activated tokens. The expert with the largest number of tokens is placed first. The goal is to make the computational load of each card as equal as possible. According to the difference between the token computational load of each card and the average value Avg, the input activation of the sliced (Slice) experts is adjusted. In addition, in the embodiment of this application, the combination of expert replicas and heterogeneous offloading is adopted to perform calculations, inter-node communication, and host-device data transfer simultaneously, further reducing the video memory occupancy of the computing card and masking the load overhead.

[0135] The embodiment of this application is mainly applied when the end-side small language model (SLM) receives complex tasks or long-sequence inference requests. In this case, the cloud needs to assist the end side to improve the inference latency and the inference effect.

[0136] As shown in Figure 3, it is the system architecture of the expert parallel processing system provided by the embodiment of this application. First, the text data "Before my bed a pool of light, I wonder if it's frost aground. The sun along the mountain bows; The Yellow River seawards flows.", "The wild prairie grass withered and flourished in turn, The boundless plain meets the sky low, The river is clear and the moon is near." is obtained through reduction (AllReduce). The text data is divided into 40 tokens through the gating network. Next, rearrangement is performed for 4 experts to determine the tokens corresponding to each expert. Through slicing (Slice), the 40 tokens are redistributed to 4 cards. Each card includes multiple experts replicated from the original experts. After the experts are redistributed, all the experts on the 4 cards and the 40 tokens are used for inference to obtain the inference result, and the inference result is collected (Allgather).

[0137] In the embodiment of this application, the expert copy deployment method is adopted. According to the hardware topology, the experts are replicated and distributed on multiple cards. For example, in the following solution, an example is given according to two copies being distributed on two cards. For example, the card can be a single Ascend computing card during the inference process. Specifically, the method provided by the embodiment of this application is illustrated as follows and mainly includes the following steps:

[0138] S1. For the data of the non-MoE part, all the data is obtained through AllReduce.

[0139] Among them, the MoE model is not entirely composed of the MoE part, and also contains the non-MoE part. The result obtained after calculation by the non-MoE part is called the calculation data of the non-MoE part. In the embodiment of the application, the data of the non-MoE part is obtained, and the data of the non-MoE part is AllReduced and then passed down.

[0140] S2. Using Gating, calculate the expert corresponding to each token.

[0141] In the embodiment of the application, a routing division algorithm is used, the experts are sorted in descending order of the number of activated tokens, the expert with the most tokens is discharged first, and the goal is to make the calculation amount of each card as equal as possible. According to the difference between the token calculation amount of each card and the average value Avg, the input of the expert is sliced (sliced), wherein discharging means arranging and placing, first selecting the current expert according to the number of tokens, placing the expert with the most token selection in the front, placing the expert with the second most token selection in the second front position, and so on. Then, according to the arrangement order of the experts, the frontmost expert is arranged first, and the token distribution to all experts is completed in this way. For example, there are 4 cards, each card is placed with an expert, assuming that card 1 needs to calculate 100 tokens, card 2 needs to calculate 300 tokens, card 3 needs to calculate 100 tokens, and card 4 needs to calculate 100 tokens, and the average value Avg is 150 tokens per card.

[0142] S3. Rearranging the tokens, and synchronously deploying the experts to different cards according to the rearrangement result.

[0143] In the embodiment of the application, distributed parallel is used when deploying experts, and each expert is deployed on two cards when dividing experts. Specifically, the experts are sorted in descending order of the number of activated tokens, and the sorted experts are [T0, T1, T2…], the cards are sorted according to the card identifier rank_id, if T0 >= Avg, the corresponding rank is placed in Avg, and the remaining part is placed in the replicated expert, and the goal is to make the calculation amount of each card equal. The Avg is the average number of activated tokens of all experts.

[0144] S4. Calculate the feedforward layer FFN part.

[0145] After the step S3 is executed, the experts are placed on each card, that is, the placing process is executed through the step 3, and then the FFN process is executed according to the experts and the token on the card, the experts and the token are calculated, and thus the reasoning result can be obtained.

[0146] In the expert deployment, the embodiment of the application adopts the multi-copy expert deployment. When the experts are divided, each expert is deployed in two cards in the form of copy, and the target is to make the calculation amount of each card tend to be equal. For example, as shown in FIG. 4, through the routing division, the number of tokens activated by the expert 0, the expert 1 and the expert 2 is 0, and the number of tokens activated by the expert 3 is 40. For the extreme case of 0, 0, 0 and 40, through the multi-copy expert deployment, the expert 3 is deployed on the card 2 and the card 3. At this time, the 40 tokens are evenly distributed to the expert 3 on the two cards. Therefore, the calculation amount on the original single card can be shared to the balanced calculation amount on the two cards, and the processing efficiency of the card is improved.

[0147] For example, the embodiment of the application can adopt the following expert division method for the multi-copy expert deployment. Ti represents the number of tokens on each expert i, Ei'rank_0 represents the first card on which the expert copy is placed, i represents the card on which the i-th expert is placed, and Ei'rank_1 represents the second card on which the expert copy is placed. The following process code is executed:

[0148] In the above expert deployment process, the sum of all the token quantities to be calculated is sorted to obtain [T0, T1, T2…], and then the average value Avg is calculated. If Ti >= Avg*2, the tokens are evenly distributed on two cards. If Avg*2 > Ti >= Avg, Avg tokens are distributed on the first card, and the rest are distributed on the second card. If Ti < Avg, the Ti corresponding tokens are distributed on the first card and the second card, so as to realize the balanced distribution.

[0149] For example, the process of deploying experts to different cards is illustrated as follows. Taking FIG. 3 as an example, 4 experts, namely expert 0, expert 1, expert 2 and expert 3, are stored in the memory of the host node. Among them, expert 0 is allocated to 8 tokens, expert 1 is allocated to 5 tokens, expert 2 is allocated to 16 tokens, and expert 3 is allocated to 11 tokens. Here, there are a total of 8+5+16+11=40 tokens, and on average, each card needs to calculate 10 tokens. Different from the prior art in which 1 expert is placed on each card, in the embodiment of the present application, 2 experts are placed on each card. For example, the 0th card is placed with expert 0 and expert 1, and it is specified that expert 0 on the 0th card only processes 5 tokens and expert 1 only processes 5 tokens. In this way, the number of tokens processed on each card is as equal as possible.

[0150] As can be known from the above example, in the embodiment of the present application, the problem of unbalanced expert load in MoE model inference is solved, the precision is not damaged, no additional calculation and communication amount is added, and in the distributed inference based on expert parallelism, the inference time delay can be improved. Other parallel methods can be used together. In addition, in the embodiment of the present application, the Offload technology can be used in combination for the experts that have been calculated to improve the computing efficiency of the display memory.

[0151] Next, the improvement of the inference performance of the large language model in the embodiment of the present application is described. As shown in Table 1 below, the MoE structure of the MoE model with a weight of 38B performs FFN calculation, and the inference time delay of the MoE structure accounts for about 54% of the total inference time delay. Table 1 below is derived from the inference execution time of each operator counted by the profiling tool on the Ascend server. When the load is unbalanced, the communication time of expert parallelism and all-to-all communication (AllToAll) will be further lengthened, and the embodiment of the present application can shorten the inference time delay of the MoE structure by about half. Therefore, for the same request, the end-to-end time delay of the distributed inference, the overall inference performance of the MoE structure in the embodiment of the present application can be improved by 1.3-1.6 times.

[0152] Table 1 is the inference execution time of each operator counted by the profiling tool on the Ascend server

[0153] In Table 1, Device_id represents the identification of the computing card, OP Type represents the computing type, Core Type represents the processor core type of the computing card, Count represents the total number of operators, Total Time (us) represents the delay of all operators to perform inference, Min Time (us) represents the minimum delay of a single operator to perform inference, Avg Time (us) represents the average delay of all operators to perform inference, Max Time (us) represents the maximum delay of a single operator to perform inference, and Ratio (%) represents the proportion of the inference delay of one computing type in the total inference delay.

[0154] Next, an expert parallel deployment process of an actual MoE model is exemplified.

[0155] In actual inference, the MoE model has multiple parallel modes, such as tensor model TP parallel and expert EP parallel. When using EP parallel, different experts are allocated to different cards. During inference, different tokens selected by different experts are inconsistent, some experts are allocated more tokens, and some experts are allocated less tokens, thereby causing load balancing problems. Embodiments of the present application optimize the load imbalance phenomenon under expert parallel through expert replicas and dynamic routing. At the same time, the Offload technology of expert weight is used to optimize the additional memory occupation problem under multiple expert replicas.

[0156] As shown in FIG. 5, it is the network architecture of a standard MoE model. The network architecture of the MoE model in actual deployment includes multi-card and expert parallel distribution. The network architecture of the MoE model includes N+1 layers, which are Layer0, …, LayerN. Each layer includes an MoE part and a non-MoE part. The MoE part is deployed with four experts, which are Expert0, Expert1, Expert2 and Expert3. The non-MoE part includes attention (Attention) data.

[0157] As shown in FIG. 6, multiple expert replicas are deployed using expert parallel. On each card, for each layer, one expert is placed. For each layer, four experts are evenly placed on one card, and four experts generated by expert replication are also evenly placed on one card. For example, Expert0 and Expert1 are placed on 0 card, Expert1 and Expert2 are placed on 1 card, Expert2 and Expert3 are placed on 2 card, and Expert3 and Expert0 are placed on 3 card. FIG. 6 shows that two experts are placed on each card, but it is not limited. Three or more experts can also be placed on each card. In embodiments of the present application, the number of experts placed on each card can also be dynamically selected.

[0158] As shown in FIG. 7, the expert copy causes additional occupation of the video memory, and only copying the experts requires twice the amount of video memory. To address the additional occupation of the video memory, the weight offload technology can be used to effectively reduce the occupation of the video memory. That is, the expert weights that are not used temporarily are offloaded to the memory of the host. When it is about to be used, preloading is performed. The calculation and reading time is masked. The calculation and reading time refers to reading the data of the next layer when calculating a layer in the calculation process. Therefore, the calculation and reading can be performed simultaneously, so that the calculation and reading time is masked.

[0159] In FIG. 7, the solid line box of Layer 0 is the expert loaded in the video memory, and the experts of layers 1 to N are temporarily placed in the memory of the host node, for example, the memory of the host node can be the memory of the server. When the 0th layer is calculated, the 1st layer of 8 experts is loaded in the video memory, and the 8 experts are 2 copies, each of which is 4 experts. The layer is numbered from 0, and when a layer is calculated, the weight of the next layer is loaded, for example, when the 1st layer is calculated, the weight of the 2nd layer is loaded, when the 8th layer is calculated, the weight of the 9th layer is loaded, and so on. When the 1st layer is calculated, the weight of the 0th layer expert is discarded, and the weight of the 2nd layer expert is loaded, and so on. In this way, the occupation of the video memory is expanded from the initial occupation W, W is the number of experts, to 2*W, and then reduced to 2*W / N, N is the number of layers. Compared with the initial scheme, the occupation of the video memory is reduced to (2 / N)W.

[0160] Based on the foregoing examples, the embodiments of the present application can solve the problem of unbalanced expert load in the MoE model reasoning, and the precision is not damaged, and the calculation amount and the communication amount are not additionally increased. In the MoE reasoning process, when the experts are parallel, the expert copy can be used, and a reasonable routing division algorithm is designed to improve the load balancing capability of the MoE calculation. The expert parallel mode can be used in parallel with the large language model and the data parallel. In the distributed reasoning based on the expert parallel, the reasoning delay can be improved. The other parallel modes can be used in stack. In the embodiments of the present application, the expert offload technology can be used to improve the efficiency of the video memory and improve the reasoning performance.

[0161] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action order described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0162] To facilitate better implementation of the above-mentioned scheme of the embodiments of the present application, the related device for implementing the above-mentioned scheme is also provided below.

[0163] Please refer to FIG. 8, the expert parallel processing system 800 provided by the embodiments of the present application can include a host node 801 and N computing cards 802, W experts are stored in the host node, the W experts are deployed from the host node to the N computing cards by expert replication, each computing card of the N computing cards includes at least two experts with different expert weights, and W and N are positive integers. The expert parallel processing system 800 performs the method shown in the foregoing FIG. 2.

[0164] Please refer to FIG. 9, the host node provided by the embodiments of the present application includes:

[0165] The acquisition module 901 is configured to acquire M text processing units token in a large language model, and M is a positive integer.

[0166] The routing division module 902 is configured to perform routing division on the M text processing units to obtain text processing units activated by the W experts respectively.

[0167] The distribution module 903 is configured to distribute the M text processing units to at least two experts on each computing card of the N computing cards according to the text processing units activated by the W experts respectively, to obtain text processing units activated by at least two experts on each computing card of the N computing cards respectively, and the total number of text processing units activated by at least two experts corresponding to the same expert weight on different computing cards is equal to the number of text processing units corresponding to the same expert weight in the W experts obtained by routing division.

[0168] In the embodiments of the present application, the module is an example of a software functional unit, and the data processing device can include code running on a computing instance. The computing instance can be at least one of a physical host (computing device), a virtual machine, a container, and the like. Further, the above-mentioned computing device can be one or more. For example, the data processing device can include code running on multiple hosts / virtual machines / containers.

[0169] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, it makes the at least one computing device execute the data processing method.

[0170] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium or data storage device that can be accessed by a computing device and includes one or more of a memory, a floppy disk, a DVD, a Blu-ray disc, a Compact Disc Read Only Memory (CD-ROM), a Compact Disc Read-Write Memory (CD-R / W), and a semiconductor memory, etc. The computer readable storage medium includes instructions that instruct the computing device to perform the data processing method.

[0171] The embodiments of the present application further provide a chip system. The chip system includes a processor configured to implement the steps performed by the computing device cluster. In a possible design, the chip system can further include a memory configured to store necessary program instructions and data. The chip system can be composed of a chip or include the chip and other discrete devices.

[0172] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other means. For example, the device embodiments described above are only schematic, and the division of units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0174] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0175] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0176] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the various embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. [Corrected according to Rule 91 on 11.09.2025] A method of processing in parallel by experts, characterized in that, The method is applied to an expert parallel processing system, and the expert parallel processing system comprises a host node and N computing cards, W experts are stored in the host node, and W and N are positive integers; The method comprises: The W experts are deployed from the host node to the N computing cards in an expert replication manner, and each of the N computing cards comprises at least two experts with different expert weights; M text processing units in a large language model are obtained, and M is a positive integer; The M text processing units are routed and divided to obtain text processing units activated by the W experts respectively; According to the text processing units activated by the W experts respectively, the M text processing units are distributed to at least two experts on each of the N computing cards to obtain text processing units activated by at least two experts on each of the N computing cards respectively, wherein the total number of text processing units activated by at least two experts corresponding to the same expert weight on different computing cards is equal to the number of text processing units corresponding to the same expert weight in the W experts obtained by routing and dividing.

2. The method of claim 1, wherein, The method further comprises: Inference is performed using at least two experts on each of the N computing cards and text processing units activated by the at least two experts on each of the N computing cards to obtain an output result of the large language model.

3. The method according to claim 1 or 2, characterized in that, The W experts are deployed from the host node to the N computing cards in an expert replication manner, comprising: W experts stored in the memory of the host node are obtained; The W experts are deployed from the memory of the host node to N video memories of the N computing cards in an expert replication manner.

4. The method according to any one of claims 1 to 3, characterized in that, According to the text processing units activated by the W experts respectively, the M text processing units are distributed to at least two experts on each of the N computing cards to obtain text processing units activated by at least two experts on each of the N computing cards respectively, comprising: The number of average expert-activated text processing units is obtained according to the text processing units activated by the W experts respectively; According to the number of average expert-activated text processing units, the number of text processing units corresponding to the same expert weight in the W experts obtained by routing and dividing is distributed to at least two experts corresponding to the same expert weight on different computing cards.

5. The method of claim 4, wherein, According to the number of average expert-activated text processing units, the number of text processing units corresponding to the same expert weight in the W experts obtained by routing and dividing is distributed to at least two experts corresponding to the same expert weight on different computing cards, comprising: A first text processing unit corresponding to a first expert is determined from the text processing units activated by the W experts respectively, the first expert is an expert in the W experts whose number of activated text processing units is greater than the number of average expert-activated text processing units, and the first expert runs on a first computing card of the N computing cards; The first T text processing units in the first text processing unit are assigned to a first expert on the first computing card, and the other text processing units in the first text processing unit except the T text processing units are assigned to a second expert on a second computing card in the N computing cards, the first expert and the second expert correspond to the same expert weight, and T represents the average number of activated text processing units of the expert.

6. The method according to any one of claims 1 to 5, characterized in that, The W experts respectively activate the text processing units, which are sorted in descending order according to the number of text processing units activated by the experts.

7. The method according to any one of claims 1 to 6, characterized in that, The W experts are deployed from the host node to the N computing cards in the form of expert replication, including: Obtain the i-th layer in the large language model, where i is a positive integer; The W experts corresponding to the i-th layer are deployed from the host node to the N computing cards in the form of expert replication, wherein the W experts corresponding to the i+j-th layer in the large language model are stored in the host node, and j is a positive integer.

8. The method of claim 7, wherein, According to the W experts respectively activated text processing units, the M text processing units are assigned to at least two experts on each computing card in the N computing cards, including: According to the W experts respectively activated text processing units corresponding to the i-th layer, the M text processing units are assigned to at least two experts on each computing card in the N computing cards. The method further comprises: After the M text processing units are assigned to at least two experts on each computing card in the N computing cards, in the process of using at least two experts on each computing card in the N computing cards for reasoning, the W experts corresponding to the i+j-th layer are deployed from the host node to the N computing cards in the form of expert replication.

9. The method of any one of claims 7-8, wherein, The method further comprises: Release the W experts that have calculated the M text processing units in the i-th layer of the large language model.

10. The method of claim 9, wherein, The method further comprises: Deploy the released W experts from the host node back to the N computing cards.

11. A processing system for experts in parallel, characterized by, The expert parallel processing system comprises a host node and N computing cards, the W experts are stored in the host node, and the W and the N are positive integers; The host node comprises: A deployment module for deploying the W experts from the host node to the N computing cards in the form of expert replication, each computing card in the N computing cards comprising at least two experts with different expert weights; An acquisition module for acquiring M text processing units token in a large language model, where M is a positive integer; A routing division module for routing division of the M text processing units to obtain text processing units respectively activated by the W experts; allocating the M text processing units to at least two experts on each of the N computing cards according to the text processing units activated by the W experts respectively, to obtain at least two text processing units activated by experts on each of the N computing cards respectively, wherein the total number of the text processing units activated by at least two experts corresponding to the same expert weight on different computing cards is equal to the number of the text processing units corresponding to the same expert weight in the W experts obtained by the routing division. 12.An electronic device comprising a host node and N computing cards, the host node storing W experts, the W experts being deployed to the N computing cards by expert replication, each of the N computing cards comprising at least two experts with different expert weights, W and N being positive integers. The host node is configured to perform the method of any one of claims 1 to 10. 13.A computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 10. 14.A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 10.

15. A chip, characterized by comprising one or more interface circuits and one or more processors; the interface circuit is configured to receive a signal from a memory of the electronic device and send the signal to the processor, the signal comprising computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method of any one of claims 1 to 10.

Images