Deployment methods, devices, electronic equipment, and storage media for hybrid expert models
By deploying hybrid expert models hierarchically on hardware devices and optimizing internal parameters, the problems of hardware utilization and training of hybrid expert models are solved, improving computing performance and load balancing, and reducing operating costs and storage requirements.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2026-04-03
AI Technical Summary
Hybrid expert models cannot fully utilize bandwidth and computing power on hardware devices, resulting in low performance. Furthermore, the training process suffers from issues such as unbalanced expert load and high operating costs.
By deploying the multi-level architecture of the hybrid expert model in hardware devices in a hierarchical manner from high to low, optimizing internal parameters by combining load balancing loss and hardware equalization loss, and saving KV cache storage requirements during the training and inference phases, a compressed multi-head attention mechanism and a multi-level hierarchical hybrid expert layer structure are adopted.
It optimizes the use of hardware bandwidth, improves the computational performance and load balancing of hybrid expert models, and reduces operating costs and storage requirements.
Smart Images
Figure CN119167988B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a method, apparatus, electronic device, and storage medium for deploying a hybrid expert model. Background Technology
[0002] With the widespread application of large-scale models in online services, such as search engines, chatbots, and virtual assistants, their high operating costs have become a significant obstacle. Hybrid expert models, with their sparsity, good model performance, and efficient inference speed, are attracting increasing attention.
[0003] However, training hybrid expert models brings new challenges. Since the architecture of hybrid expert models is often large, the bandwidth and computing power of hardware devices cannot be fully utilized when using hybrid expert models, resulting in low performance. Summary of the Invention
[0004] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for deploying hybrid expert models, so as to improve the computational performance of expert models.
[0005] In a first aspect, embodiments of this application provide a method for deploying a hybrid expert model, wherein the hybrid expert model includes multiple transformer modules, and each transformer module includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-level hierarchical hybrid expert layer;
[0006] The compressed multi-head attention mechanism layer includes a codec, a multi-head attention mechanism module, and a position encoder;
[0007] The multi-level hierarchical hybrid expert layer includes a multi-level hybrid expert architecture;
[0008] The deployment method includes:
[0009] Obtain the hardware device to be deployed, which includes multiple layers;
[0010] The multi-level hybrid expert architecture is deployed in the corresponding levels of the hardware device from high to low; wherein the level of the hybrid expert architecture deployed in the lower-level hardware device is no higher than the level of the hybrid expert architecture deployed in the higher-level hardware device.
[0011] This application embodiment optimizes the use of hardware bandwidth at different levels in the cluster by deploying the hardware device-based hierarchy and the hybrid expert architecture hierarchy, thereby improving the computational performance of the hybrid expert model.
[0012] In any embodiment, the method further includes:
[0013] Obtain training samples, which include question-answer pairs consisting of question statements and answer statements;
[0014] The query statements in the training samples are input into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained.
[0015] Calculate the prediction loss based on the prediction results and the corresponding response statements to the query statements;
[0016] The actual number of times each expert model is used in the hybrid expert model to be trained in the statistical batch training is calculated, and the expected usage frequency of each expert model is calculated.
[0017] Calculate the load balancing loss based on the actual number of uses and the expected usage frequency;
[0018] The internal parameters of the hybrid expert model to be trained are optimized based on the prediction loss and the load balancing loss to obtain a trained hybrid expert model.
[0019] In this embodiment of the application, when training a multi-level hybrid expert model, the internal parameters are optimized by combining load balancing loss and prediction loss, so that the activation probability of each expert model in the optimized model is more balanced, thereby improving the load balancing of the expert model.
[0020] In any embodiment, optimizing the internal parameters of the hybrid expert model to be trained based on the prediction loss and the load balancing loss includes:
[0021] Obtain hardware equalization loss and communication equalization loss on different levels of hardware devices;
[0022] The internal parameters of the hybrid expert model to be trained are optimized based on the hardware balancing loss, the communication balancing loss, the prediction loss, and the load balancing loss on the different hardware devices.
[0023] In this embodiment of the application, when training a multi-level hybrid expert model, the internal parameters are optimized by combining hardware balancing loss, communication balancing loss, load balancing loss and prediction loss, so that the optimized model has improvements in terms of load balancing and computing performance of the expert model.
[0024] In any embodiment, the step of inputting the query statements from the training samples into the hybrid expert model to be trained, and obtaining the prediction result output by the hybrid expert model to be trained, includes:
[0025] The query statements in the training samples are input into the hybrid expert model to be trained. The compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the query statements to obtain the first hidden layer features.
[0026] Upsample the first hidden layer features to obtain features of the same size as the input query statement, and then delete the first hidden layer features;
[0027] The features are input into the multi-level hierarchical hybrid expert layer to obtain the prediction result.
[0028] In this embodiment, the first hidden layer features are not saved during the model training phase, thus saving the storage requirements of the KV cache.
[0029] In any embodiment, the method further includes:
[0030] Receive a query statement and input the query statement into the hybrid expert model;
[0031] The query statement is normalized by the normalization layer to obtain a normalized query statement;
[0032] The normalized query statement is encoded and decoded using the encoder and decoder in the compressed multi-head attention mechanism layer to obtain the second hidden layer features, and the query vector, key vector and value vector are obtained based on the second hidden layer features;
[0033] The query vector, key vector, and value vector are processed using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain the processing result;
[0034] The processing result is further processed using the multi-level hierarchical hybrid expert layer to obtain the query result corresponding to the query statement.
[0035] In this embodiment, the query vector, key vector, and value vector are not saved during the inference stage. Instead, the query vector, key vector, and value vector are recovered through the second hidden layer features, thus saving the storage requirements of the KV cache.
[0036] In any embodiment, the normalization layer includes a first normalization module and a second normalization module;
[0037] The first normalization module is positioned before the compressed multi-head attention mechanism layer;
[0038] The second normalization module is positioned after the compressed multi-head attention mechanism layer.
[0039] In any embodiment, each level of the hybrid expert architecture includes a shared expert module and a non-shared expert module;
[0040] The number of levels in the hierarchical hybrid expert layer, as well as the total number of shared and non-shared expert modules, are determined by the model sparsity ratio.
[0041] The embodiments of this application determine the number of hierarchical hybrid expert layers and the total number of shared and non-shared expert modules by the model sparsity ratio, thereby reducing the complexity and computational cost of the model.
[0042] Secondly, embodiments of this application provide a deployment apparatus for a hybrid expert model, wherein the hybrid expert model includes multiple transformer modules, and each transformer module includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-level hierarchical hybrid expert layer;
[0043] The compressed multi-head attention mechanism layer includes a codec, a multi-head attention mechanism module, and a position encoder;
[0044] The multi-level hierarchical hybrid expert layer includes a multi-level hybrid expert architecture;
[0045] The deployment device includes:
[0046] The acquisition module is used to acquire the hardware device to be deployed, which includes multiple layers;
[0047] The deployment module is used to deploy the multi-level hybrid expert architecture in the corresponding levels of the hardware device from high to low; wherein the level of the hybrid expert architecture deployed in the lower-level hardware device is no higher than the level of the hybrid expert architecture deployed in the higher-level hardware device.
[0048] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a bus, wherein,
[0049] The processor and the memory communicate with each other via the bus;
[0050] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.
[0051] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium, comprising:
[0052] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of the first aspect.
[0053] Fifthly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method of the first aspect.
[0054] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description
[0055] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0056] Figure 1 A schematic diagram of the architecture of a transformer module provided in an embodiment of this application;
[0057] Figure 2 A structural diagram of a compressed multi-head attention mechanism provided in an embodiment of this application;
[0058] Figure 3 A multi-level hierarchical hybrid expert layer structure diagram provided for embodiments of this application;
[0059] Figure 4 A schematic flowchart illustrating a method for deploying a hybrid expert model, as provided in an embodiment of this application;
[0060] Figure 5 A schematic diagram of a deployment device for a hybrid expert model provided in an embodiment of this application;
[0061] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0062] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.
[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.
[0064] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.
[0065] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0066] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.
[0067] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).
[0068] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.
[0069] With the widespread application of large-scale models in online services, such as search engines, chatbots, and virtual assistants, their high operating costs have become a significant obstacle. Hybrid expert models, with their sparsity, good model performance, and efficient inference speed, have attracted increasing attention. However, training hybrid expert models brings new challenges, such as: unbalanced expert load; the requirement for lossless training; and the all-to-all bandwidth requirements of hybrid expert models for the cluster.
[0070] Mistral AI has open-sourced its large language model based on hybrid expert models, and deepSeek has open-sourced their work on hybrid expert models. Addressing the training challenges of hybrid expert models, Microsoft, Google, and others have published their experiences in MOE training, such as communication and computation overlap and different types of parallel strategies. Long context inference is a challenge for large models due to its storage and computation complexity and the inherent O(N) complexity of the context length N. 2 Understanding the relationship between context and compression is of great significance.
[0071] Through long-term research, the inventors of this application discovered that when deploying hybrid expert models in a cluster, the bandwidth varies between different servers within the cluster due to the hierarchical structure of the hardware devices, and also between different cards within a single service. Furthermore, the hybrid expert models themselves also exist in a hierarchical manner, with different levels of expert models having different bandwidth requirements. Therefore, to improve the computational performance of hybrid expert models, when deploying them, the expert models are deployed on hardware devices corresponding to their respective hierarchical levels. For example, the highest-level expert model is deployed on the highest-level hardware device, and lower-level expert models are deployed on relatively lower-level hardware devices.
[0072] This application provides a hybrid expert model, which includes multiple transformer modules. Each transformer module includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-level hierarchical hybrid expert layer (multi-H-MOE). Figure 1 This application provides an example of an architecture diagram of a transformer module, as shown below. Figure 1As shown. The normalization norm layer can be RMS, lay norm, or other normalization methods, and the norm layer position can include Pre-Norm and Post-Norm. Pre-Norm refers to performing normalization first, followed by function mapping. Post-Norm refers to performing function mapping first, followed by normalization. In this embodiment, Pre-Norm can be set before the compressed multi-head attention mechanism layer, and Post-Norm can be set after the compressed multi-head attention mechanism layer.
[0073] The hybrid expert model proposed in this application can achieve scaling effects by changing parameters such as the number of stacked transformer modules and the hidden dimension inside the transformer module.
[0074] The compressed-attention layer consists of an encoder-decoder, a multi-head attention module, and a position encoder (ROPE). Figure 2 A compressed multi-head attention mechanism structure diagram is provided for embodiments of this application, such as... Figure 2 As shown:
[0075] The encoder-decoder is used to encode and decode the input tokens.
[0076] The multi-head attention module aims to capture information from different angles and levels in an input sequence by processing multiple independent attention heads in parallel. This allows the model to abstract the sequence from multiple different representation spaces, thereby improving the model's expressive and learning capabilities. Its working principle is as follows: For the input sequence, multiple query, key, and value matrices are first generated through linear transformations. Each head has its own independent linear transformation matrix. Each head independently executes the attention mechanism, calculating the dot product (or other similarity measure) between the query and key, then obtaining attention weights through a softmax function, and finally using these weights to perform a weighted summation of the values. The outputs of all heads are fused through methods such as concatenation or averaging to form a unified representation. The fused representation undergoes another linear transformation to obtain the final output of the multi-head attention module.
[0077] The position encoder ROPE can incorporate the position information of tokens. In large language models, a sentence composed of the same words in different orders may have completely different meanings. Therefore, by incorporating position information, the hybrid expert model can more accurately understand the meaning expressed by the input sentence.
[0078] Figure 3 A multi-level hierarchical hybrid expert layer structure diagram is provided for embodiments of this application, such as... Figure 3 As shown, the multi-level hierarchical hybrid expert layer includes a multi-level hybrid expert MOE architecture. Figure 3 The diagram shows three levels. In practical applications, the number of levels can be reduced or increased according to actual needs, and this application does not specifically limit this. The MOE architectures of different levels are isomorphic. Each level of the MOE architecture contains a non-shared expert model and may also contain a shared expert model. Furthermore, the shared and non-shared expert models are constructed based on a top-K expert network of the router. That is, based on the characteristics of the input data, the probability or weight of each expert being selected is calculated. Based on the output of the routing function, the K experts with the highest probability or weight are selected, and the input data or its features are divided into K parts, with each part sent to one of the selected experts.
[0079] In a shared expert model, a subset of experts (i.e., neural network modules) are shared and activated when processing all inputs. These shared experts are responsible for capturing and integrating common knowledge or general information from different contexts. By using shared experts, knowledge redundancy between parameters of independent routing experts (i.e., non-shared experts) can be reduced, improving parameter efficiency. Because some experts are shared, it is not necessary to recalculate the outputs of these shared experts when processing new inputs, thus reducing computational costs.
[0080] Non-shared expert models: In non-shared expert models, all experts are independent and are selectively activated based on the input. This design allows each expert to focus on processing specific inputs or tasks. Because each expert is independent, different experts can be flexibly combined and activated as needed to handle different inputs or tasks. Furthermore, each non-shared expert can focus on a specific domain or task, thereby exhibiting higher performance in that domain or task.
[0081] Figure 4 This is a schematic diagram of a deployment method for a hybrid expert model provided in an embodiment of this application, as shown below. Figure 4 As shown, the method includes:
[0082] Step 401: Obtain the hardware device to be deployed, which includes multiple layers;
[0083] Step 402: Deploy the multi-level hybrid expert architecture in the corresponding levels of the hardware devices in descending order of hierarchy; wherein the level of the hybrid expert architecture deployed in the lower-level hardware devices is no higher than the level of the hybrid expert architecture deployed in the higher-level hardware devices.
[0084] In the specific implementation process, before deploying the trained hybrid expert model to the cluster, it is necessary to determine which hardware devices in the cluster should be used to deploy the hybrid expert model; that is, to obtain the hardware devices to be deployed. The hardware devices are also hierarchical. Specifically, the cluster consists of multiple servers that communicate with each other via switches. Each server contains multiple GPUs, and each GPU includes multiple cards. Therefore, the hierarchy from high to low can be divided into: switch-server-GPU. The hybrid expert model provided in this embodiment adopts a three-level MOE architecture. Each level of the MOE architecture contains 2 shared experts and 16 non-shared experts, forming a top-4 expert network. Approximately 16... 3 = 4096 experts, each token has (2+4) 3 =216 experts participated in the calculation, with a sparsity ratio of approximately 5%. The multi-level MOE architecture can adapt to the cluster interconnection hierarchy to achieve efficient use of cluster bandwidth. For example, in the first-level multi-H-MOE, all parameters are on nodes at the same level; in the second-level multi-H-MOE, all parameters are on servers at the same level; and in the third-level multi-H-MOE, all parameters are on GPUs at the same level, etc., thereby optimizing the use of hardware bandwidth at different levels in the cluster. Any hardware or system with a hierarchical structure can deploy multi-H-MOE at different levels; it is even possible to deploy the hierarchical experts in the multi-H-MOE proposed in this invention on different stream processing units (SMs) in the GPU.
[0085] Therefore, the level of hybrid expert architecture deployed in low-level hardware devices is no higher than the level of hybrid expert architecture deployed in high-level hardware devices.
[0086] This application embodiment optimizes the use of hardware bandwidth at different levels in the cluster by deploying the hardware device-based hierarchy and the hybrid expert architecture hierarchy, thereby improving the computational performance of the hybrid expert model.
[0087] Based on the above embodiments, the method further includes:
[0088] Obtain training samples, which include question-answer pairs consisting of question statements and answer statements;
[0089] The query statements in the training samples are input into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained.
[0090] Calculate the prediction loss based on the prediction results and the corresponding response statements to the query statements;
[0091] The actual number of times each expert model is used in the hybrid expert model to be trained in the statistical batch training is calculated, and the expected usage frequency of each expert model is calculated.
[0092] Calculate the load balancing loss based on the actual number of uses and the expected usage frequency;
[0093] The internal parameters of the hybrid expert model to be trained are optimized based on the prediction loss and the load balancing loss to obtain a trained hybrid expert model.
[0094] In the specific implementation process, the hybrid expert model can be trained in advance before deployment. Taking the application scenario corresponding to the hybrid expert model protected in the embodiments of this application as a question-answering model as an example, the following description is made.
[0095] First, a large number of training samples are obtained, each consisting of a question and an answer, forming a question-answer pair. These training samples are then divided into multiple parts for batch training of the hybrid expert model.
[0096] The query statements from the training samples are input into the hybrid expert model to be trained. The hybrid expert model to be trained analyzes and processes the query statements and outputs the prediction results.
[0097] The prediction loss is calculated based on the prediction results and the corresponding responses to the queries.
[0098] The actual usage frequency of each expert model in the batch training of the hybrid expert model to be trained is counted, and the expected usage frequency of each expert model is calculated. The expert models include shared expert models and non-shared expert models. The expected usage frequency of each expert model is calculated based on an ideal uniform distribution.
[0099] The difference between the actual usage frequency and the target distribution is used as the load balancing loss. This difference is typically measured using the Kullback-Leibler divergence.
[0100] L balance =D KL (P actual ||P target )
[0101] Among them, P actual It is the actual usage frequency, P target It is the target usage frequency, i.e., the uniformly distributed frequency.
[0102] The load balancing loss and the prediction loss during model training are combined to form the total loss. The internal parameters of the hybrid expert model to be trained are then optimized based on the total loss to complete a batch of training.
[0103] Once the training termination condition is met, the last batch of hybrid expert models trained is used as the trained hybrid expert model. The training termination condition can be either that the number of training batches reaches a preset number, or that the rate of change of the training loss value is less than a preset value.
[0104] In this embodiment of the application, when training a multi-level hybrid expert model, the internal parameters are optimized by combining load balancing loss and prediction loss, so that the activation probability of each expert model in the optimized model is more balanced, thereby improving the load balancing of the expert model.
[0105] Based on the above embodiments, the optimization of the internal parameters of the hybrid expert model to be trained according to the prediction loss and the load balancing loss includes:
[0106] Obtain hardware equalization loss and communication equalization loss on different levels of hardware devices;
[0107] The internal parameters of the hybrid expert model to be trained are optimized based on the hardware balancing loss, the communication balancing loss, the prediction loss, and the load balancing loss on the different hardware devices.
[0108] In practical implementation, to ensure balanced training load, multi-level MOE adds balancing between different hardware (nodes or different computing cards) and between different hardware during training. This avoids over-utilization of any particular hardware or communication method, which could affect overall efficiency. Therefore, hardware balancing loss and communication balancing loss are added to the model loss function. Both hardware balancing loss and communication balancing loss can be defined using KL divergence to measure the distance between the actual hardware and communication usage and the frequency distribution of their corresponding average usage.
[0109] The sum of the hardware balancing loss, the communication balancing loss, the prediction loss, and the load balancing loss on different hardware devices is used as the total loss. Based on the total loss, the internal parameters of the hybrid expert model to be trained are optimized to complete a batch of training.
[0110] This application's embodiments improve model training accuracy by adding expert load balancing losses at different levels of MOE (Model-Oriented Environment) to the traditional large-scale model task loss during training; load balancing losses across different computing devices; hardware balancing losses across different computing nodes; and communication balancing losses. A tiered token-dropping strategy, such as token-dropping strategies on different nodes and on different servers, is employed to achieve smaller token-dropping losses and tiered balancing losses, thereby improving the accuracy of model training.
[0111] Based on the above embodiments, the step of inputting the query statements from the training samples into the hybrid expert model to be trained, and obtaining the prediction results output by the hybrid expert model to be trained, includes:
[0112] The query statements in the training samples are input into the hybrid expert model to be trained. The compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the query statements to obtain the first hidden layer features.
[0113] Upsample the first hidden layer features to obtain features of the same size as the input query statement, and then delete the first hidden layer features;
[0114] The features are input into the multi-level hierarchical hybrid expert layer to obtain the prediction result.
[0115] In the specific implementation process, when training the hybrid expert model to be trained, the compressed multi-head attention mechanism layer in the hybrid expert model to be trained processes the received input data as follows:
[0116] After inputting the query statements from the training samples into the hybrid expert model to be trained, if the first layer of the hybrid expert model is a normalization layer, then the query statements are normalized through the normalization layer. If the first layer is a compressed multi-head attention mechanism layer, then the compressed multi-head attention mechanism layer processes the query statements.
[0117] Therefore, the input data in this embodiment refers to the data input to the compressed multi-head attention mechanism layer, which may be a query statement or a normalized result.
[0118] The Compressed-Attention layer uses an encoder-decoder based encoder to encode and decode the input tokens. Before the multi-head-attention module in the original transformer, the Compressed-Attention layer adds an encoder-decoder. This encoder-decoder first downsamples to form pyramidal hidden features of different levels (i.e., the first hidden feature latent). For example, if the length of the input first hidden feature is 5210, after three downsampling operations, the lengths of the first hidden features are 256, 128, and 64 respectively. These three pyramidal downsampled first hidden features are concatenated to form a 256 + 128 + 64 = 448-layer hidden feature vector. Then, an upsampling method is used to restore the encoder-decoder feature vector length of 5210. This encoding and decoding process uses a low-rank method to reduce the number of weight parameters. The number of encoder-decoder layers and the dimension of the first hidden feature are determined by the expected KV cache compression ratio of the model. This embodiment uses low-rank compression: the KV cache is compressed into a low-rank latent vector through the encoding layer, thereby reducing storage requirements. During the inference phase, this compression significantly reduces access to the KV cache, accelerating processing. Assuming the original hidden layer feature dimension is d, the encoding / decoding hidden layer feature dimension is r, and the KV cache compression ratio is [compression ratio], the specific expression is as follows:
[0119]
[0120] The encoder-decoder encodes and decodes the input tokens, implementing a hidden layer representation of the input features and compressing the KV cache storage requirements. During model training, the Compressed-Attention layer does not need to save the first hidden layer features (Latent), i.e., the first hidden layer features are deleted.
[0121] The features obtained after upsampling are input into a multi-level hierarchical hybrid expert layer, which analyzes them to obtain prediction results.
[0122] In this embodiment, the first hidden layer features are not saved during the model training phase, thus saving the storage requirements of the KV cache.
[0123] Based on the above embodiments, the method further includes:
[0124] Receive a query statement and input the query statement into the hybrid expert model;
[0125] The query statement is normalized by the normalization layer to obtain a normalized query statement;
[0126] The normalized query statement is encoded and decoded using the encoder and decoder in the compressed multi-head attention mechanism layer to obtain the second hidden layer features, and the query vector, key vector and value vector are obtained based on the second hidden layer features;
[0127] The query vector, key vector, and value vector are processed using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain the processing result;
[0128] The processing result is further processed using the multi-level hierarchical hybrid expert layer to obtain the query result corresponding to the query statement.
[0129] In the specific implementation process, when using the trained hybrid expert model for inference, a query statement is received and input into the hybrid expert model. First, the query statement is normalized by the normalization layer in the hybrid expert model to obtain a normalized query statement. Then, the normalized query statement is input into a compressed multi-head attention mechanism layer. The encoder and decoder in the compressed multi-head attention mechanism layer encodes and decodes the normalized query statement to obtain the second hidden layer features, and based on the second hidden layer features, the query vector, key vector, and value vector are obtained. The multi-head attention mechanism module processes the query vector, key vector, and value vector to obtain the processing results. Each head independently executes the attention mechanism, calculates the dot product (or other similarity measure) between the query and the key, and then obtains the attention weights through the softmax function. These weights are then used to perform a weighted summation of the value. The outputs of all heads are fused through concatenation or averaging to form a unified representation. The fused representation is then subjected to a linear transformation to obtain the final output of Multi-Head Attention. Then, the final output of Multi-HeadAttention is input into a multi-level hierarchical hybrid expert layer. Based on the input result, the multi-level hierarchical hybrid expert layer matches k expert models and activates these k expert models. These expert models then process the input result to obtain the final query result.
[0130] It should be noted that since a hybrid expert model includes multiple transformer modules, the input of the second transformer module is the output of the first transformer module, and so on, until the last transformer module outputs the final query result.
[0131] In this embodiment, the Compressed-Attention layer saves on KV cache storage requirements by storing the latent features of the hidden QKV layer instead of the KV cache during the model inference stage.
[0132] Based on the above embodiments, each level of the hybrid expert architecture includes a shared expert module and a non-shared expert module;
[0133] The number of levels in the hierarchical hybrid expert layer, as well as the total number of shared and non-shared expert modules, are determined by the model sparsity ratio.
[0134] In practical implementation, the sparsity ratio refers to the probability that each input sample is assigned to an expert. It determines the model's complexity and computational cost. A lower sparsity ratio leads to more input samples being assigned to experts, increasing model complexity and computational cost; while a higher sparsity ratio reduces these issues. The number of experts and the top_K parameter of an efficient hybrid expert model can be determined based on the sparsity ratio.
[0135] For example, the hybrid expert model in this embodiment has an L-level hybrid expert architecture, with s shared expert models and m non-shared expert models in each level. Each time, k non-shared expert models are selected for activation from the m non-shared models. Therefore, the sparsity of this hybrid expert model is:
[0136]
[0137] Knowing the sparsity, adjust the top-k parameters and the number of shared and non-shared expert models to ensure that the sparsity ratio of the final model meets the requirements.
[0138] This application's embodiment utilizes a stacked multi-transformer module approach, resulting in a scheme with less information redundancy, finer-grained experts, more balanced model training loss, and more efficient inference deployment. MOE training often requires all-to-all bandwidth; this application's embodiment fully leverages the interconnect bandwidth between different layers of the training cluster to improve MOE training efficiency.
[0139] It should be noted that the hybrid expert model provided in this application embodiment can be implemented through YARN and other methods to achieve long context models, and can be combined with reinforcement learning and Monte Carlo tree search and other applications.
[0140] The embodiments of this application determine the number of hierarchical hybrid expert layers and the total number of shared and non-shared expert modules by the model sparsity ratio, thereby reducing the complexity and computational cost of the model.
[0141] Figure 5 This is a schematic diagram of a deployment device for a hybrid expert model provided in an embodiment of this application. The device can be a module, program segment, or code on an electronic device. It should be understood that this device is similar to the one described above. Figure 4 The method implementation corresponds to this and can be executed. Figure 4 The specific functions of the device involved in the various steps of the method embodiment can be found in the description above. To avoid repetition, detailed descriptions are omitted here. The hybrid expert model includes multiple transformer modules, and each transformer module includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-level hierarchical hybrid expert layer.
[0142] The compressed multi-head attention mechanism layer includes a codec, a multi-head attention mechanism module, and a position encoder;
[0143] The multi-level hierarchical hybrid expert layer includes a multi-level hybrid expert architecture;
[0144] The deployment device includes:
[0145] The acquisition module 501 is used to acquire the hardware device to be deployed, which includes multiple layers;
[0146] The deployment module 502 is used to deploy the multi-level hybrid expert architecture in the corresponding levels of the hardware device from high to low; wherein the level of the hybrid expert architecture deployed in the lower-level hardware device is not higher than the level of the hybrid expert architecture deployed in the higher-level hardware device.
[0147] Based on the above embodiments, the device further includes a training module for:
[0148] Obtain training samples, which include question-answer pairs consisting of question statements and answer statements;
[0149] The query statements in the training samples are input into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained.
[0150] Calculate the prediction loss based on the prediction results and the corresponding response statements to the query statements;
[0151] The actual number of times each expert model is used in the hybrid expert model to be trained in the statistical batch training is calculated, and the expected usage frequency of each expert model is calculated.
[0152] Calculate the load balancing loss based on the actual number of uses and the expected usage frequency;
[0153] The internal parameters of the hybrid expert model to be trained are optimized based on the prediction loss and the load balancing loss to obtain a trained hybrid expert model.
[0154] Based on the above embodiments, this training module is specifically used for:
[0155] Obtain hardware equalization loss and communication equalization loss on different levels of hardware devices;
[0156] The internal parameters of the hybrid expert model to be trained are optimized based on the hardware balancing loss, the communication balancing loss, the prediction loss, and the load balancing loss on the different hardware devices.
[0157] Based on the above embodiments, this training module is specifically used for:
[0158] The query statements in the training samples are input into the hybrid expert model to be trained. The compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the input data to obtain the first hidden layer features.
[0159] Upsample the first hidden layer features to obtain features of the same size as the input query statement, and then delete the first hidden layer features;
[0160] The features are input into the multi-level hierarchical hybrid expert layer to obtain the prediction result.
[0161] Based on the above embodiments, the device further includes a reasoning module for:
[0162] Receive a query statement and input the query statement into the hybrid expert model;
[0163] The query statement is normalized by the normalization layer to obtain a normalized query statement;
[0164] The normalized query statement is encoded and decoded using the encoder and decoder in the compressed multi-head attention mechanism layer to obtain the second hidden layer features, and the query vector, key vector and value vector are obtained based on the second hidden layer features;
[0165] The query vector, key vector, and value vector are processed using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain the processing result;
[0166] The processing result is further processed using the multi-level hierarchical hybrid expert layer to obtain the query result corresponding to the query statement.
[0167] Based on the above embodiments, the normalization layer includes a first normalization module and a second normalization module;
[0168] The first normalization module is positioned before the compressed multi-head attention mechanism layer;
[0169] The second normalization module is positioned after the compressed multi-head attention mechanism layer.
[0170] Based on the above embodiments, each level of the hybrid expert architecture includes a shared expert module and a non-shared expert module;
[0171] The number of levels in the hierarchical hybrid expert layer, as well as the total number of shared and non-shared expert modules, are determined by the model sparsity ratio.
[0172] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of this application, such as... Figure 6 As shown, the electronic device includes: a processor 601, a memory 602, and a bus 603; wherein,
[0173] The processor 601 and the memory 602 communicate with each other through the bus 603;
[0174] The processor 601 is used to call program instructions in the memory 602 to execute the methods provided in the above method embodiments, such as: obtaining a hardware device to be deployed, the hardware device including multiple levels; deploying the multi-level hybrid expert architecture in the corresponding levels of the hardware device from high to low; wherein the level of the hybrid expert architecture deployed in the lower-level hardware device is not higher than the level of the hybrid expert architecture deployed in the higher-level hardware device.
[0175] Processor 601 can be an integrated circuit chip with signal processing capabilities. The processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.
[0176] The memory 602 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.
[0177] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute the methods provided in the above-described method embodiments, such as: acquiring a hardware device to be deployed, the hardware device including multiple levels; deploying the multi-level hybrid expert architecture in the corresponding levels of the hardware device according to the levels from high to low; wherein the level of the hybrid expert architecture deployed in the lower-level hardware device is not higher than the level of the hybrid expert architecture deployed in the higher-level hardware device.
[0178] This embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause the computer to execute the methods provided in the above-described method embodiments. These instructions include, for example,: acquiring a hardware device to be deployed, the hardware device comprising multiple layers; deploying the multi-level hybrid expert architecture in the corresponding layers of the hardware device according to the layers from high to low; wherein the layer of the hybrid expert architecture deployed in the lower-level hardware device is no higher than the layer of the hybrid expert architecture deployed in the higher-level hardware device.
[0179] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0180] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0181] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0182] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.
[0183] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A method for deploying a hybrid expert model, characterized in that, The hybrid expert model includes multiple transformer modules, each transformer module including a normalization layer, a compressed multi-head attention mechanism layer, and a multi-level hierarchical hybrid expert layer; The compressed multi-head attention mechanism layer includes a codec, a multi-head attention mechanism module, and a position encoder; The multi-level hierarchical hybrid expert layer includes a multi-level hybrid expert architecture; The deployment method includes: Obtain the hardware device to be deployed, which includes multiple layers; The multi-level hybrid expert architecture is deployed in the hardware device in descending order of hierarchy; wherein the level of the hybrid expert architecture deployed in the lower-level hardware device is no higher than the level of the hybrid expert architecture deployed in the higher-level hardware device. The method further includes: Obtain training samples, which include question-answer pairs consisting of question statements and answer statements; The query statements in the training samples are input into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained. Calculate the prediction loss based on the prediction results and the corresponding response statements to the query statements; The actual number of times each expert model is used in the hybrid expert model to be trained in the statistical batch training is calculated, and the expected usage frequency of each expert model is calculated. Calculate the load balancing loss based on the actual number of uses and the expected usage frequency; The internal parameters of the hybrid expert model to be trained are optimized based on the prediction loss and the load balancing loss to obtain a trained hybrid expert model.
2. The method according to claim 1, characterized in that, The optimization of the internal parameters of the hybrid expert model to be trained based on the prediction loss and the load balancing loss includes: Obtain hardware equalization loss and communication equalization loss on different levels of hardware devices; The internal parameters of the hybrid expert model to be trained are optimized based on the hardware balancing loss, the communication balancing loss, the prediction loss, and the load balancing loss on the different hardware devices.
3. The method according to claim 1, characterized in that, The step of inputting the query statements from the training samples into the hybrid expert model to be trained, and obtaining the prediction results output by the hybrid expert model to be trained, includes: The query statements in the training samples are input into the hybrid expert model to be trained. The compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the input data to obtain the first hidden layer features. Upsample the first hidden layer features to obtain features of the same size as the input query statement, and then delete the first hidden layer features; The features are input into the multi-level hierarchical hybrid expert layer to obtain the prediction result.
4. The method according to claim 3, characterized in that, The method further includes: Receive a query statement and input the query statement into the hybrid expert model; The query statement is normalized by the normalization layer to obtain a normalized query statement; The normalized query statement is encoded and decoded using the encoder and decoder in the compressed multi-head attention mechanism layer to obtain the second hidden layer features, and the query vector, key vector and value vector are obtained based on the second hidden layer features; The query vector, key vector, and value vector are processed using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain the processing result; The processing result is further processed using the multi-level hierarchical hybrid expert layer to obtain the query result corresponding to the query statement.
5. The method according to claim 1, characterized in that, The normalization layer includes a first normalization module and a second normalization module; The first normalization module is positioned before the compressed multi-head attention mechanism layer; The second normalization module is positioned after the compressed multi-head attention mechanism layer.
6. The method according to any one of claims 1-5, characterized in that, Each level of hybrid expert architecture includes shared expert modules and non-shared expert modules; The number of levels in the hierarchical hybrid expert layer, as well as the total number of shared and non-shared expert modules, are determined by the model sparsity ratio.
7. A deployment device for a hybrid expert model, characterized in that, The hybrid expert model includes multiple transformer modules, each transformer module including a normalization layer, a compressed multi-head attention mechanism layer, and a multi-level hierarchical hybrid expert layer; The compressed multi-head attention mechanism layer includes a codec, a multi-head attention mechanism module, and a position encoder; The multi-level hierarchical hybrid expert layer includes a multi-level hybrid expert architecture; The deployment device includes: The acquisition module is used to acquire the hardware device to be deployed, which includes multiple layers; The deployment module is used to deploy the multi-level hybrid expert architecture in the corresponding levels of the hardware device from high to low; wherein the level of the hybrid expert architecture deployed in the lower-level hardware device is no higher than the level of the hybrid expert architecture deployed in the higher-level hardware device. The deployment device further includes a training module for: Obtain training samples, which include question-answer pairs consisting of question statements and answer statements; The query statements in the training samples are input into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained. Calculate the prediction loss based on the prediction results and the corresponding response statements to the query statements; The actual number of times each expert model is used in the hybrid expert model to be trained in the statistical batch training is calculated, and the expected usage frequency of each expert model is calculated. Calculate the load balancing loss based on the actual number of uses and the expected usage frequency; The internal parameters of the hybrid expert model to be trained are optimized based on the prediction loss and the load balancing loss to obtain a trained hybrid expert model.
8. An electronic device, characterized in that, include: Processor, memory, and bus, among which, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-6 by calling the program instructions.
9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-6.
10. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-6.
Citation Information
Patent Citations
Model training method, image classification method, server and storage medium
CN114170425A
Window-level hybrid expert model and generation method thereof
CN115310457A