Lightweight hybrid expert model architecture system and implementation method thereof

Abstract

The present invention discloses a lightweight hybrid expert model architecture system and an implementation method thereof, which are corresponding schemes. In the schemes: dimension reduction and dimension increase operations are introduced before and after communication respectively, and the traditional communication-computation-communication mode is transformed into a new mode of dimension reduction-communication-computation-communication-dimensional increase, which can reduce the content volume of data migration communication, thereby effectively reducing communication overhead; and the model architecture provided by the present invention can be divided according to the parallel scheme during deployment, and deployed using various training frameworks, so as to efficiently perform reasoning and training tasks on a GPU (graphics processing unit).

Description

Technical Field

[0001] The present invention relates to the technical field of hybrid expert model design, and in particular to a lightweight hybrid expert model architecture system and an implementation method thereof. Background Art

[0002] The Mixture of Experts (MoE) model architecture is recognized as an efficient model expansion technology in the field of machine learning because it can linearly expand the model size while only incurring sub-linear additional computing overhead. In early large language model (LLM) applications, the MoE structure mainly linearly expands the feed forward network (FFN) layer and combines the routing mechanism to select appropriate experts to participate in the calculation, resulting in significant data migration communication (All to All Communication) overhead during the training process.

[0003] Recently, the DeepSeek-V2 large language model proposed by the DeepSeek team uses a fine-grained mixture of experts (Fine Grained MoE) structure. By adjusting the size of the matrix dimensions involved in the FFN operation, the original large FFN structure is replaced with a larger and lighter FFN structure, achieving performance that is no less than the original model. However, as the development trend of MoE is moving towards a larger number of experts and a smaller scale, the number of experts for routing selection (top k ) also increased, which resulted in a large amount of data migration communication overhead. MoE model training and inference tests conducted on four servers equipped with eight NVIDIA A40 GPUs (a graphics processor launched by NVIDIA) showed that more than 50% of the time was spent on data migration communication for both, and this proportion of time increased as the top k rising with the growth of

[0004] Therefore, in order to meet this challenge, it is necessary to propose a new MoE model architecture, which aims to significantly reduce a large amount of data migration communication and thus improve the efficiency of model training and reasoning. Summary of the invention

[0005] The purpose of the present invention is to provide a lightweight hybrid expert model architecture system and its implementation method, which can reduce the content volume of data migration communication, thereby effectively reducing communication overhead and further improving the efficiency of model training and reasoning.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A lightweight hybrid expert model architecture system, comprising: a large language model with a lightweight hybrid expert structure configured at each layer; the lightweight hybrid expert structure comprises: a threshold network, a dimensionality reduction operation unit, multiple experts, an output combination unit and a dimensionality increase operation unit; wherein:

[0008] The threshold network determines the input word metadata and determines the corresponding relationship between the word metadata and each expert;

[0009] The dimension reduction operation unit performs a dimension reduction operation on the word metadata and transmits the reduced-dimensional word metadata to the corresponding expert;

[0010] After receiving the reduced word metadata, each expert performs parallel processing;

[0011] The output combination unit combines the results of the parallel processing of each expert and outputs the result;

[0012] The dimension-raising operation unit performs a dimension-raising operation on the output of the output combination unit to obtain an output result with the same dimension as the input word data.

[0013] A method for implementing a lightweight hybrid expert model architecture comprises: configuring a lightweight hybrid expert structure at each layer in a large language model; the lightweight hybrid expert structure comprises: a threshold network, a dimensionality reduction operation unit, multiple experts, an output combination unit and a dimensionality increase operation unit; wherein:

[0014] The threshold network determines the input word metadata and determines the corresponding relationship between the word metadata and each expert;

[0015] The dimension reduction operation unit performs a dimension reduction operation on the word metadata and transmits the reduced-dimensional word metadata to the corresponding expert;

[0016] After receiving the reduced word metadata, each expert performs parallel processing;

[0017] The output combination unit combines the results of the parallel processing of each expert and outputs the result;

[0018] The dimension-raising operation unit performs a dimension-raising operation on the output of the output combination unit to obtain an output result with the same dimension as the input word data.

[0019] It can be seen from the technical solution provided by the present invention that by introducing dimensionality reduction and dimensionality increase operations before and after communication respectively, the traditional communication-computation-communication mode can be transformed into a new mode of dimensionality reduction-communication-computation-communication-dimensionality increase, which can reduce the content volume of data migration communication, thereby effectively reducing communication overhead; and the architecture provided by the present invention can be divided according to the parallel scheme at the time of deployment, and deployed using various training frameworks, so as to efficiently perform reasoning and training tasks on a GPU (graphics processing unit). BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0021] Figure 1 A schematic diagram of a lightweight hybrid expert model architecture system provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the existing coarse-grained MoE model architecture provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of an existing fine-grained MoE model architecture provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of the MoE model architecture of the present invention provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of tensor parallelism and sequence parallelism of an existing MoE model provided in an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of a parallel structure solution adopted by the NVIDIA Megatron-LM framework provided in an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of a data parallelization solution provided by an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of a parallelization scheme for tensor parallelism and sequence parallelism provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0030] First, the terms that may be used in this article are explained as follows:

[0031] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0032] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0033] The following is a detailed description of a lightweight hybrid expert model architecture system provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If no specific conditions are specified in the embodiments of the present invention, they shall be carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer.

[0034] Embodiment 1

[0035] The embodiment of the present invention provides a lightweight hybrid expert model architecture system, including: a large language model with a lightweight hybrid expert (MoE) structure configured at each layer, such as Figure 1 As shown, the lightweight hybrid expert structure (i.e., MoE structure) mainly includes: a threshold network, a dimensionality reduction operation unit, multiple experts, an output combination unit, and a dimensionality increase operation unit; wherein: the threshold network discriminates the input word metadata and determines the corresponding relationship between the word metadata and each expert; the dimensionality reduction operation unit performs dimensionality reduction operation on the word metadata and transmits the reduced word metadata to the corresponding expert; each expert performs parallel processing after receiving the reduced word metadata; the output combination unit combines the results of the parallel processing of each expert and outputs them; the dimensionality increase operation unit performs dimensionality increase operation on the output of the output combination unit to obtain an output result with the same dimension as the input word metadata.

[0036] In one embodiment, the dimension reduction operation unit performs a dimension reduction operation on the word data, including: using a shared dimension reduction matrix to perform a dimension reduction operation on the word data, thereby reducing the amount of data required for data migration communication; the dimension increase operation unit performs a dimension increase operation on the output of the output combination unit, including: using a shared dimension increase matrix to perform a dimension increase operation on the output of the output combination unit to restore the data dimension. The sharing means that each node uses the same parameters for dimension increase and dimension reduction in distributed deployment.

[0037] In one embodiment, the output combination unit can combine the results of parallel processing by each expert (i.e., the output of each expert) in a weighted summation manner, that is, a weight is assigned to each expert, the expert's output is weighted using the weight, and then the synthesis of the weighted results is calculated to obtain the combined output.

[0038] The large language model described in the embodiment of the present invention can be any existing large language model, for example, the DeepSeek-V2 large language model described in the above background technology; in addition, the large language model belongs to the technical terminology in this field, generally refers to a model with more than one billion parameters, which is not described in detail in the present invention. For ease of description, the large language model with a lightweight mixture of experts (MoE) structure configured at each layer is referred to as the MoE model architecture.

[0039] The solution provided by the embodiment of the present invention not only improves communication efficiency, but also maintains low additional resource consumption, providing a more economical and efficient solution for the implementation of large-scale machine learning tasks.

[0040] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0041] 1. Overall overview of the plan.

[0042] The embodiment of the present invention provides a lightweight MoE structure, which aims to significantly reduce the data migration communication overhead in machine learning tasks. To achieve this goal, the present invention introduces a pair of shared dimensionality reduction matrices and dimensionality increase matrices in the expert structure in the MoE structure, and by adjusting the order of communication and calculation, the traditional communication-computation-communication mode is transformed into a new mode of dimensionality reduction-communication-computation-communication-dimensionality increase. This innovative method can linearly reduce the content volume of data migration communication, thereby effectively reducing communication overhead.

[0043] Specifically, the present invention first performs a dimensionality reduction operation to reduce the amount of data required for data migration and communication, then performs calculations through experts, and finally restores the data dimension through a dimensionality increase operation. The above method not only optimizes the communication and calculation processes, but also introduces shared parameters that only bring very little video memory overhead (less than 2%) and calculation overhead (less than 5%), ensuring the efficiency and practicality of the model.

[0044] 2. Detailed introduction of the plan.

[0045] Under the premise of making minor adjustments to the model structure, this invention innovatively proposes a new communication strategy (low-dimensional communication strategy) and designs an efficient expert structure, which can be combined with the existing MoE optimization technology to effectively improve the throughput of training and reasoning.

[0046] 1. Low-dimensional communication strategy.

[0047] In order to intuitively understand the innovation of the present invention, the existing fine-grained and coarse-grained MoE models are introduced below.

[0048] like Figure 2 The figure shows the existing coarse-grained MoE structure, which has certain challenges in communication strategy. In this structure, data is first routed through a threshold network (not shown in the figure) to determine which word metadata (token) will be passed to the corresponding experts for calculation. The calculation process involves one dimensional matrix operation and one dimensional matrix operation, and the number of experts involved in the calculation is determined by the parameter top k The parallel design of this MoE structure is more complicated because it is usually based on the top1 (i.e., each word is processed by the best expert) or top2 (i.e., each word is processed by the best two experts), which limits the improvement of parallel efficiency. Figure 2 In is the dimension-raising matrix among experts, is the dimension reduction matrix among experts.

[0049] To overcome these challenges, DeepSeek-V2 large language model proposes a new architecture, namely Figure 3 The fine-grained MoE structure shown in Figure 1 uses smaller experts and increases the number of experts. In order to facilitate unified understanding, Figure 3 The dimension-raising matrix and dimension-reducing matrix in the MoE structure provided by the present invention also use these symbols and However, the specific form and content of the dimension-up and dimension-down matrices in different MoE structures can be adjusted by the user according to the actual situation. This new architecture shows significant improvements in model quality and computational efficiency. However, despite the improvements in MoE structure design, this new structure still faces serious communication problems when it is actually deployed.

[0050] In order to improve computing efficiency and solve the problem of single hardware accelerator video memory limitation, a common solution is to use Expert Parallelism (EP) to distribute multiple experts in the MoE structure on different GPUs. However, when using parallelism to train and reason about the MoE structure, since the device where the current token is located may be inconsistent with the device where the target expert is located, a large amount of communication overhead will be generated for exchanging token data, which leads to a significant communication bottleneck. The size of the communication data volume C can be expressed as:

[0051] ;

[0052] Among them, ep represents the degree of parallelism of the expert parallel, b represents the number of requests that make up a batch in the inference or training task, s represents the number of tokens in each request, and h represents the hidden dimension of the large language model. These communication overheads themselves take up most of the time in the training process, and they will also increase as the top k With the improvement of the design performance, it is expected that the proportion of communication overhead will increase further as more and more expert designs are presented. Therefore, optimization is urgently needed.

[0053] To this end, an embodiment of the present invention proposes a low-dimensional communication strategy, which is based on the existing fine-grained MoE structure and aims to reduce the size of the communication content. The traditional communication-computation-communication strategy is manifested in the existing fine-grained MoE structure as a matrix operation of communication-dimensionality reduction-dimensionality increase-communication. This solution causes data migration communication to always process high-dimensional data, thereby generating a large amount of communication overhead. In order to solve this problem, in the MoE structure provided by the present invention, a shared dimensionality reduction matrix is ​​introduced before communication. Figure 4 As shown, the token data is first passed through this shared dimension reduction matrix before being passed to the expert for calculation Then, at the end of the parallel computation, a shared dimension-enhanced matrix is ​​added to restore the original data dimension. . This strategy ensures that the amount of data processed during the communication process is greatly reduced, thereby effectively reducing the communication overhead. Theoretically, assuming that the shared dimension reduction matrix compresses the original data dimension to t times the original (where t<1), then the communication volume of data migration communication will also become t times the original. This compression method is theoretically very efficient because it not only reduces the size of the communication data, but also maintains the accuracy of the calculation results.

[0054] Figure 2 N in the above table refers to the number of experts. K Indicates that each word is processed by the best K experts. Figure 3 and Figure 4 The 4N in the above is the number of experts, Top 4K It means that each word is processed by the best 4K experts, that is, the number of experts and the number of optimal experts selected are Figure 2 4 times the number of experts in the system; the specific values ​​of N and K involved here can be set according to actual conditions, and the present invention does not limit them; Figure 3 and Figure 4 In Chinese: C stands for Compress, which means compression; D stands for Descending, which means dimensionality reduction; A stands for Ascending, which means dimensionality increase.

[0055] 2. Expert structure design.

[0056] The expert structure in the MoE structure can be mathematically expressed as:

[0057] ;

[0058] in, represents the activation function, h is the hidden dimension of the large language model, is the hidden dimension of the expert in the MoE structure. Based on the low-dimensional communication scheme in the above embodiment, the expert design in the original fine-grained MoE structure needs to be further improved. Otherwise, if the original dimension reduction-dimensionality increase calculation method is continued, the parameter h will be significantly reduced due to the existence of the increased dimension reduction matrix, resulting in a large reduction in the number of model parameters compared to the original, thereby damaging the model performance. Figure 4 As shown, in order to align the size of the present invention and the existing fine-grained MoE structure, the hidden dimension of the expert in the MoE structure of the present invention should be increased. From the shape of the model, the expert design in the MoE structure of the present invention is very similar to the expert design in the existing coarse-grained MoE structure. The expert structure (i.e., the processing process of the expert) provided by the embodiment of the present invention is:

[0059] ;

[0060] in, is the processing result of the i-th expert, is the expert’s input data (it is the word metadata after dimensionality reduction), is the dimension-raising matrix of the i-th expert, is the dimension reduction matrix of the i-th expert.

[0061] In order to illustrate the superiority of the present invention, a comparison is made through three metrics, as shown in Table 1.

[0062] Table 1: Differences between the MoE model architecture of the present invention and the existing MoE model architecture in terms of parameter quantity, computational complexity, and communication complexity

[0063]

[0064] As shown in Table 1, GPT-Fine-Grained is the existing MoE model architecture, which is a large language model using a fine-grained MoE structure. GPT-BigMac is the MoE model architecture of the present invention, which is a large language model using the MoE structure provided by the present invention. Compared with GPT-Fine-Grained, the number of parameters of the model of GPT-BigMac is only increased by , the model's FLOPS (floating point operations per second) indicator has only increased , and the communication overhead is compressed to r times of the original, where r represents the ratio of the dimension in the expert structure to the data dimension of the MoE structure input, which can represent the degree of compression of the communication volume of data migration communication after adopting the MoE structure of the present invention. Represents the number of model layers, and s represents the sequence length of the input sample. For example, the data after the actual deployment t=0.25 corresponding to Table 1 shows that the number of parameters (i.e., Param) of the MoE model architecture of the present invention is only 1.35% more than the original one, the amount of calculation (i.e., FLOPS) is only 4.54% more than the original one, and the amount of communication (i.e., A2A) is reduced by 75% compared with the original one. This design enables the MoE model architecture of the present invention to maintain the number of parameters and calculation complexity similar to the existing MoE model architecture while significantly reducing the amount of communication.

[0065] Among them, the calculation formulas for the parameter amount, calculation amount and communication amount of the model architecture are shown in Table 2.

[0066] Table 2: Calculation formulas for the number of parameters, computation, and communication of the model architecture

[0067]

[0068] In addition to high communication efficiency, the MoE model architecture provided by the embodiment of the present invention also has other advantageous features. First, the new structure of the present invention supports a routing strategy that does not discard tokens (Dropless). In the original MoE model architecture training and reasoning process, in order to reduce the phenomenon of communication imbalance, the expert's ability parameter is usually introduced to determine the upper limit of the number of tokens accepted by each expert. When an expert exceeds the upper limit of the number of tokens accepted by the expert, the excess tokens will be discarded. Later, the DeepSeek-V2 large language model believed that canceling this parameter restriction could enable the model to be more fully trained, but at the same time, it would also introduce the maximum communication overhead. However, the natural communication efficiency of the MoE model provided by the embodiment of the present invention will be more conducive to this non-discarding routing design. Secondly, some studies believe that top k The larger the choice of , the better the performance of the model may be. Obviously, top k The increase in the top will lead to a larger amount of communication data, which will bring serious communication pressure under the original model design. However, under the MoE model design provided by the embodiment of the present invention, a more flexible top k choice.

[0069] 3. Overall model architecture implementation plan.

[0070] Based on the introduction of the above two parts, the MoE structure provided by the embodiment of the present invention mainly includes: a threshold network, a dimensionality reduction operation unit, multiple experts, an output combination unit and a dimensionality increase operation unit; wherein, the threshold network determines which word data (token) will be passed to the corresponding expert for calculation, and the dimensionality reduction operation unit performs dimensionality reduction operation on the word data through a shared dimensionality reduction matrix, and then transmits it to the corresponding expert. The expert is mainly responsible for the relevant calculation and processing process, and the output combination unit is responsible for combining the outputs of the experts. Finally, the dimensionality increase operation unit performs dimensionality increase operation on the output of the output combination unit through the dimensionality increase matrix.

[0071] 4. Parallelization scheme of model architecture during deployment.

[0072] When the MoE model architecture provided by the embodiment of the present invention is deployed on an accelerator card for training or reasoning tasks, the original parallel method may no longer be applicable in some scenarios due to the introduction of a new shared matrix, so the parallelization scheme needs to be readjusted. The embodiment of the present invention will design the parallelization scheme of the model architecture from the perspectives of data parallelism (DP) and tensor parallelism (TP).

[0073] Those skilled in the art will understand that, in actual deployment, the entire model is deployed, that is, the MoE model architecture described above. However, the main innovation of this application is to provide a new MoE structure. Therefore, the parallelization scheme will be mainly introduced in conjunction with the MoE structure in the following text.

[0074] The MoE model architecture provided by the embodiment of the present invention can be applicable to various existing training frameworks, such as the NVIDIA Megatron-LM framework, Tutel, etc.; NVIDIA Megatron-LM is an open source deep learning framework released by NVIDIA, which is specially designed for training and optimizing ultra-large-scale language models; Tutel is an optimization framework designed for the MoE training system, which can dynamically switch parallel strategies during each iteration to cope with computing requirements under different input data distributions.

[0075] The following uses the NVIDIA Megatron-LM framework as an example to illustrate.

[0076] like Figure 5 As shown, the parallel optimization scheme for the existing coarse-grained MoE structure will introduce two pairs of data migration communications and a pair of Reduce Scatter (reduced scatter) operations and All Gather (collected all) operations. Reduce Scatter refers to the operation of summing and then dividing the partial data held by all nodes after distributed computing, and All Gather refers to the operation of integrating and splicing the partial data held by all nodes after distributed computing. The nodes refer to the various GPU devices participating in the distributed computing (specifically, the experts described in the text); SP-TP means dividing the data from the SP form to the TP form through one communication, and similarly, TP-SP means dividing the data from the TP form to the SP form through one communication, SP means performing the corresponding All Gather or Reduce Scatter operation on the data in the SP form, TP means performing the corresponding All Gather or Reduce Scatter operation on the data in the TP form, and EP means that the corresponding communication process adopts the EP method. First, the NVIDIA Megatron-LM framework has a feature in its implementation process, that is, its sequence parallelism (SP) and tensor parallelism are bound together, and these two parallel methods have completely different ways of splitting data. SP requires data to be split based on the sequence or single token in each batch, while TP requires data to be split based on the internal dimension of the token. Therefore, when switching between parallel methods, it is necessary to introduce data migration communication. Figure 6As shown, the operation process of the transformer layer (Transformer Layer) of a general large language model is implemented in NVIDA Megatron-LM, wherein the layer normalization processing (LayerNorm), self-attention mechanism (Self Attention), matrix multiplication (Linear), random inactivation mechanism (Dropout), and activation function involved are all technical terms in this field. The specific implementation method can be implemented with reference to conventional technology, and the present invention will not go into details. As an example, the type of activation function can be a Gaussian linear error unit (Gelu). Indicates the All Gather operation. express The inverse operation is the Reduce Scatter operation. In order to calculate the Dropout layer and the LayerNorm layer in a parallel manner, when entering the MoE structure, the data is still divided in the form of SP, and the implementation of the MoE structure of the present invention is divided in TP mode, so at this time it is necessary to introduce data migration communication to convert the data from SP to TP form for subsequent calculations. The subsequent data migration communication is the All to All (EP) data migration communication existing in the MoE structure mentioned above, which is used to transfer data to the device where the corresponding expert is located. Since the parameters of the MoE model provided in the embodiment of the present invention are divided according to TP, that is, the matrix parameters in the model structure are evenly divided on different physical devices, there are naturally corresponding Reduce Scatter and All Gather operations.

[0077] Those skilled in the art can understand that each layer of the large language model is divided into two parts: attention mechanism and expert operation. The expert operation includes the operation of the expert layer and the regularization and normalization processing of the operation results. The specific regularization and normalization processing of the operation included in the above-mentioned subsequent operation can also represent the Attention transport of the next layer.

[0078] In order to better adapt to the MoE model architecture provided by the embodiment of the present invention, two parallelization schemes are proposed. One is to perform DP processing on the newly added shared increased-dimensional matrix and reduced-dimensional matrix respectively, and the other is to divide the newly added shared increased-dimensional matrix and reduced-dimensional matrix respectively into TP. The designs of the corresponding parallelization schemes of the two methods are completely different.

[0079] (1) A parallelization scheme for DP-ing the newly added shared up-dimensional matrix and down-dimensional matrix.

[0080] like Figure 7As shown in the figure, the newly added shared matrix performs DP processing respectively, and then performs the corresponding All Reduce operation on the gradient at the end of each iteration. Specifically, the way of performing DP processing on the newly added shared matrix is ​​equivalent to storing the first and last dimension reduction matrices and dimension increase matrices in the form of copies between various devices (GPU devices), and then performing corresponding aggregation operations on the gradients each time the gradients are updated. Because this processing does not affect the subsequent internal processing of the expert, the subsequent parallel scheme within the expert (that is, the scheme after the expert processing starts) is consistent with the original one. The advantage of this parallel scheme is that it does not introduce redundant communication operations and is highly efficient. However, if the size of the shared matrix becomes large to a certain extent, the use of tensor parallelism will save more memory and improve computational efficiency.

[0081] Comparison Figure 7 The process of this part can be described as follows: DP processing is performed on the dimension-raising matrix and the dimension-reducing matrix respectively, and the dimension of the input data is reduced by the dimension-reducing matrix after DP processing. Then, the data is converted from SP to TP form through data migration communication, and then transmitted to the corresponding experts through EP data migration communication through All Gather operation; the output of each expert is subjected to Reduce Scatter operation through the output combination unit and through EP data migration communication, and then the data is converted from TP to SP form through data migration communication to reach the dimension-raising operation unit, and the dimension-raising matrix after DP processing is subjected to dimension-raising processing to obtain the output result.

[0082] Considering that the distributed training process, the gradient updating process, and the aggregation operation can all be implemented by referring to conventional technologies, they will not be described in detail.

[0083] (2) A parallelization scheme for processing the newly added shared dimension-upgrading matrix and dimension-downgrading matrix into TPs respectively.

[0084] like Figure 8 As shown, since the data just after the Attention layer is calculated is still segmented in SP form, the Attention layer is Figure 6The Self Attention shown is the calculation part before the MoE structure in the first layer of the MoE model architecture. At this time, since the first shared matrix (i.e., the shared dimensionality reduction matrix) is segmented in TP form, an All Gather alignment operation is required in the sequence dimension for subsequent calculations. After the first shared matrix, the data is segmented in TP form. The present invention changes the logic of rotating data in the NVIDIA Megatron-LM framework, removes the original redundant communication logic that needs to transform SP segmentation into TP segmentation, and changes the data segmentation method in the expert. The original expert calculation logic can be regarded as two consecutive matrix multiplication operations. The first matrix is ​​segmented by rows and the second matrix is ​​distinguished by columns. The data obtained in this way is part of the result, and an All Gather alignment operation is performed. Reduce can get the final result. Now, since the result after passing through the shared dimensionality reduction matrix is ​​divided by rows, the first matrix in the expert is divided by columns and the second matrix is ​​divided by rows. The result of the data passing through the first matrix in the expert only needs to be aggregated to get the correct result. Similarly, the calculation and communication process of the second matrix in the expert and the shared dimensionality increase matrix is ​​similar to that of the shared dimensionality reduction matrix and the first matrix in the expert. Only one Reduce-Scatter operation is needed at the end to transform the data from TP segmentation to SP segmentation.

[0085] Comparison Figure 8 , the process of this part can be described as: TP processing is performed on the dimension-raising matrix and the dimension-reducing matrix respectively, the All Gather operation is performed on the input word metadata in the sequence dimension, and then the dimension is reduced by the dimension-reducing matrix after TP processing; and it is transmitted to the corresponding expert through the EP data migration communication; each expert processing process involves two matrices, the first matrix is ​​the dimension-raising matrix in the expert, and the second is the dimension-reducing matrix in the expert. The first matrix is ​​divided according to columns, and the second matrix is ​​divided according to rows. After the first matrix after division and the activation function processing, multiple activation outputs are obtained, all activation functions are aggregated, and then the second matrix after division is processed to obtain the output of the expert; the output of each expert is combined by the output combination unit and transmitted to the dimension-raising operation unit through the EP data migration communication, and the dimension is increased by the dimension-raising matrix after TP processing. Finally, the output result is obtained after the data is converted into SP form through the Reduce Scatter operation.

[0086] In the embodiment of the present invention, the output combination unit is responsible for all operations and communication processes between the expert output and the dimension-upgrading operation unit.

[0087] Similarly, the MoE model architecture provided by the embodiment of the present invention can also be implemented in Tutel. The results show that, at the same parameter level, the performance of the MoE model architecture provided by the embodiment of the present invention implemented in Tutel is much better than the performance of the existing MoE model architecture in Tutel.

[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0089] Embodiment 2

[0090] The present invention also provides a method for implementing a lightweight hybrid expert model architecture, which is a method corresponding to the aforementioned system, and the method mainly comprises: configuring a lightweight hybrid expert structure in each layer of a large language model; the lightweight hybrid expert structure comprises: a threshold network, a dimensionality reduction operation unit, multiple experts, an output combination unit and a dimensionality increase operation unit; wherein: the threshold network discriminates the input word metadata and determines the corresponding relationship between the word metadata and each expert; the dimensionality reduction operation unit performs a dimensionality reduction operation on the word metadata and transmits the reduced-dimensional word metadata to the corresponding expert; each expert performs parallel processing after receiving the reduced-dimensional word metadata; the output combination unit combines the results of the parallel processing of each expert and outputs them; the dimensionality increase operation unit performs a dimensionality increase operation on the output of the output combination unit to obtain an output result with the same dimension as the input word metadata.

[0091] In an embodiment of the present invention, the dimensionality reduction operation unit performs a dimensionality reduction operation on the word metadata, including: using a shared dimensionality reduction matrix to perform a dimensionality reduction operation on the word metadata; the dimensionality increase operation unit performs a dimensionality increase operation on the output of the output combination unit, including: using a shared dimensionality increase matrix to perform a dimensionality increase operation on the output of the output combination unit.

[0092] In the embodiment of the present invention, experts can be distributed on different graphics processors in an expert parallel manner. The processing process of the experts is expressed as follows:

[0093] ;

[0094] in, is the processing result of the i-th expert, is the word metadata after dimensionality reduction, is the dimension-raising matrix of the i-th expert, is the dimension reduction matrix of the i-th expert, is the activation function.

[0095] In an embodiment of the present invention, a large language model in which each layer is configured with a lightweight hybrid expert structure is applied to a training or reasoning task, and a parallelization scheme of the MoE structure is designed, including:

[0096] (1) A parallelization scheme for DP-ing an increased-dimensional matrix and a reduced-dimensional matrix, comprising the following steps: DP-ing an increased-dimensional matrix and a reduced-dimensional matrix, reducing the dimensionality of input data by using the reduced-dimensional matrix after DP-ing, and then first converting the data from SP to TP form by data migration communication, and then transmitting the data to corresponding experts through EP data migration communication through a full collection operation; the output of each expert is subjected to a reduction and dispersion operation by an output combination unit and then to EP data migration communication, and then the data is converted from TP to SP form by data migration communication and then reaches the increased-dimensional operation unit, and the increased-dimensional matrix after DP-ing is subjected to increased-dimensional processing to obtain an output result; wherein DP represents data parallelism, TP represents tensor parallelism, EP represents expert parallelism, and SP represents sequence parallelism.

[0097] (2) A parallelization scheme for TP-ing the dimension-raising matrix and the dimension-reducing matrix, comprising the following steps: TP-ing the dimension-raising matrix and the dimension-reducing matrix, respectively; performing a collection operation on the input word metadata in the sequence dimension; and then reducing the dimension through the dimension-reducing matrix after TP-ing; and transmitting the matrix to the corresponding expert through EP data migration communication; each expert processing process involves two matrices, the first matrix is ​​the dimension-raising matrix in the expert, and the second matrix is ​​the dimension-reducing matrix in the expert, the first matrix is ​​split according to columns, and the second matrix is ​​split according to rows, and multiple activation outputs are obtained after the first matrix is ​​split and the activation function is processed, all activation functions are aggregated, and then the second matrix is ​​processed after the split to obtain the output of the expert; the output of each expert is combined by the output combination unit and transmitted to the dimension-raising operation unit through EP data migration communication, and the dimension is increased by the dimension-raising matrix after TP-ing; finally, the output result is obtained after the data is converted into SP form through the reduction dispersion operation.

[0098] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0099] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A lightweight hybrid expert model architecture system, characterized in that: include: Each layer is equipped with a large language model with a lightweight hybrid expert structure; The lightweight hybrid expert structure includes: a threshold network, a dimensionality reduction operation unit, multiple experts, an output combination unit and a dimensionality increase operation unit; wherein: The threshold network determines the input word metadata and determines the corresponding relationship between the word metadata and each expert; The dimension reduction operation unit performs a dimension reduction operation on the word metadata and transmits the reduced-dimensional word metadata to the corresponding expert; After receiving the reduced word metadata, each expert performs parallel processing; The output combination unit combines the results of the parallel processing of each expert and outputs the result; The dimension-raising operation unit performs a dimension-raising operation on the output of the output combination unit to obtain an output result with the same dimension as the input word data; The method further includes: when the large language model in which each layer is configured with a lightweight hybrid expert structure is applied to a training or reasoning task, a parallelization scheme of the lightweight hybrid expert structure is designed, including: A parallelization scheme for DP-ing an increased-dimensional matrix and a decreased-dimensional matrix, comprising the following steps: DP-ing an increased-dimensional matrix and a decreased-dimensional matrix, reducing the dimensionality of input data by using the decreased-dimensional matrix after DP-ing, and then first converting the data from SP to TP form by using data migration communication, and then transmitting the data to corresponding experts through all collection operations after using data migration communication in an EP mode; the output of each expert is subjected to a reduction and dispersion operation by an output combination unit and is subjected to data migration communication in an EP mode, and then the data is converted from TP to SP form by using data migration communication and then reaches the increased-dimensional operation unit, and the increased-dimensional matrix after DP-ing is subjected to increased-dimensional processing to obtain an output result; wherein DP represents data parallelism, TP represents tensor parallelism, EP represents expert parallelism, and SP represents sequence parallelism; A parallelization scheme for TP-ing an increased-dimensional matrix and a decreased-dimensional matrix, comprising the following steps: TP-ing an increased-dimensional matrix and a decreased-dimensional matrix, respectively, performing a collection operation on all input word metadata in a sequence dimension, and then reducing the dimension through the decreased-dimensional matrix after TP-ing; and transmitting the reduced-dimensional matrix to a corresponding expert through data migration communication in an EP manner; each expert processing process involves two matrices, the first matrix is ​​the increased-dimensional matrix in the expert, and the second is the decreased-dimensional matrix in the expert, the first matrix is ​​segmented according to columns, and the second matrix is ​​segmented according to rows, and multiple activation outputs are obtained after the segmented first matrix and activation function processing, all activation functions are aggregated, and then the output of the expert is obtained through the segmented second matrix processing; the output of each expert is combined by an output combination unit and transmitted to a increased-dimensional operation unit through data migration communication in an EP manner, and the increased-dimensional matrix after TP-ing is used for dimensioning, and finally, the data is converted into an SP form through a reduction-dispersed operation to obtain an output result; wherein TP represents tensor parallelism, EP represents expert parallelism, and SP represents sequence parallelism.

2. A lightweight hybrid expert model architecture system according to claim 1, characterized in that: The dimensionality reduction operation unit performs dimensionality reduction operation on word data, including: using a shared dimensionality reduction matrix to perform dimensionality reduction operation on word data; the dimensionality increase operation unit performs dimensionality increase operation on the output of the output combination unit, including: using a shared dimensionality increase matrix to perform dimensionality increase operation on the output of the output combination unit.

3. A lightweight hybrid expert model architecture system according to claim 1, characterized in that: Also includes: The experts are distributed on different GPUs in an expert parallel manner. The expert processing process is expressed as: ； in, is the processing result of the i-th expert, is the word metadata after dimensionality reduction, is the dimension-raising matrix of the i-th expert, is the dimension reduction matrix of the i-th expert, is the activation function.

4. A method for implementing a lightweight hybrid expert model architecture, characterized in that: include: Each layer in the large language model is configured with a lightweight hybrid expert structure; The lightweight hybrid expert structure includes: a threshold network, a dimensionality reduction operation unit, multiple experts, an output combination unit and a dimensionality increase operation unit; wherein: The threshold network discriminates the input word metadata and determines the corresponding relationship between the word metadata and each expert; The dimension reduction operation unit performs a dimension reduction operation on the word metadata and transmits the reduced-dimensional word metadata to the corresponding expert; After receiving the reduced word metadata, each expert performs parallel processing; The output combination unit combines the results of the parallel processing of each expert and outputs the result; The dimension-increasing operation unit performs a dimension-increasing operation on the output of the output combination unit to obtain an output result of the same dimension as the input word data; The method further includes: when the large language model in which each layer is configured with a lightweight hybrid expert structure is applied to a training or reasoning task, a parallelization scheme of the lightweight hybrid expert structure is designed, including: A parallelization scheme for DP-ing an increased-dimensional matrix and a decreased-dimensional matrix, comprising the following steps: DP-ing an increased-dimensional matrix and a decreased-dimensional matrix, reducing the dimensionality of input data by using the decreased-dimensional matrix after DP-ing, and then first converting the data from SP to TP form by using data migration communication, and then transmitting the data to corresponding experts through all collection operations after using data migration communication in an EP mode; the output of each expert is subjected to a reduction and dispersion operation by an output combination unit and is subjected to data migration communication in an EP mode, and then the data is converted from TP to SP form by using data migration communication and then reaches the increased-dimensional operation unit, and the increased-dimensional matrix after DP-ing is subjected to increased-dimensional processing to obtain an output result; wherein DP represents data parallelism, TP represents tensor parallelism, EP represents expert parallelism, and SP represents sequence parallelism; A parallelization scheme for TP-ing an increased-dimensional matrix and a decreased-dimensional matrix, comprising the following steps: TP-ing an increased-dimensional matrix and a decreased-dimensional matrix, respectively, performing a collection operation on all input word metadata in a sequence dimension, and then reducing the dimension through the decreased-dimensional matrix after TP-ing; and transmitting the reduced-dimensional matrix to a corresponding expert through data migration communication in an EP manner; each expert processing process involves two matrices, the first matrix is ​​the increased-dimensional matrix in the expert, and the second is the decreased-dimensional matrix in the expert, the first matrix is ​​segmented according to columns, and the second matrix is ​​segmented according to rows, and multiple activation outputs are obtained after the segmented first matrix and activation function processing, all activation functions are aggregated, and then the output of the expert is obtained through the segmented second matrix processing; the output of each expert is combined by an output combination unit and transmitted to a increased-dimensional operation unit through data migration communication in an EP manner, and the increased-dimensional matrix after TP-ing is used for dimensioning, and finally, the data is converted into an SP form through a reduction-dispersed operation to obtain an output result; wherein TP represents tensor parallelism, EP represents expert parallelism, and SP represents sequence parallelism.

5. A method for implementing a lightweight hybrid expert model architecture according to claim 4, characterized in that: The dimensionality reduction operation unit performs dimensionality reduction operation on word data, including: using a shared dimensionality reduction matrix to perform dimensionality reduction operation on word data; the dimensionality increase operation unit performs dimensionality increase operation on the output of the output combination unit, including: using a shared dimensionality increase matrix to perform dimensionality increase operation on the output of the output combination unit.

6. A method for implementing a lightweight hybrid expert model architecture according to claim 4, characterized in that: Also includes: The experts are distributed on different GPUs in an expert parallel manner. The expert processing process is expressed as: ； in, is the processing result of the i-th expert, is the word metadata after dimensionality reduction, is the dimension-raising matrix of the i-th expert, is the dimension reduction matrix of the i-th expert, is the activation function.

Images