Hybrid expert model deployment method and device, electronic equipment and storage medium

Abstract

The invention provides a deployment method and device of a hybrid expert model, electronic equipment and a storage medium. The hybrid expert model comprises a plurality of transformer modules, and each transformer module comprises a normalization layer, a compressed multi-head attention mechanism layer and a multi-layer hierarchical hybrid expert layer; the compressed multi-head attention mechanism layer comprises a codec, a multi-head attention mechanism module and a position encoder; the multi-layer hierarchical hybrid expert layer comprises a multi-level hybrid expert architecture; the deployment method comprises the following steps: acquiring to-be-deployed hardware equipment of multiple hierarchies; deploying the multi-stage hybrid expert architecture in corresponding hierarchies in the hardware equipment from high to low according to the hierarchies; the hierarchy of the hybrid expert architecture deployed in the low-hierarchy hardware equipment is not higher than the hierarchy of the hybrid expert architecture deployed in the high-hierarchy hardware equipment. Through corresponding deployment based on the hierarchy of the hardware equipment and the hierarchy of the hybrid expert architecture, the use of hardware bandwidths of different hierarchies in a cluster is optimized, and the computing performance of the hybrid expert model is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a deployment method, device, electronic device and storage medium of a hybrid expert model. Background Art

[0002] With the widespread use of large models in online services such as search engines, chatbots, and virtual assistants, their high operating costs have become a significant obstacle. Hybrid expert models have attracted more and more attention due to their sparsity, good model performance, and efficient inference speed.

[0003] However, the training of hybrid expert models brings new challenges. Since the architecture of hybrid expert models is often large, when using hybrid expert models, the bandwidth and computing power of hardware devices cannot be fully utilized, resulting in lower performance. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a deployment method, device, electronic device and storage medium of a hybrid expert model to improve the computing performance of the expert model.

[0005] In a first aspect, an embodiment of the present application provides a deployment method of a hybrid expert model, wherein the hybrid expert model includes multiple transformer modules, each transformer module includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-layer hierarchical hybrid expert layer;

[0006] The compressed multi-head attention mechanism layer includes an encoder-decoder, a multi-head attention mechanism module, and a position encoder;

[0007] The multi-layer hierarchical hybrid expert layer includes a multi-level hybrid expert architecture;

[0008] The deployment method includes:

[0009] Acquire hardware equipment to be deployed, wherein the hardware equipment includes multiple levels;

[0010] The multi-level hybrid expert architecture is deployed in the corresponding level of the hardware device from high to low level; wherein the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

[0011] The embodiment of the present application optimizes the use of hardware bandwidth at different levels in the cluster by performing corresponding deployment based on the level of hardware devices and the level of the hybrid expert architecture, thereby improving the computing performance of the hybrid expert model.

[0012] In any embodiment, the method further comprises:

[0013] Acquire a training sample, wherein the training sample includes a question-answer pair consisting of a query sentence and an answer sentence;

[0014] Inputting the query sentences in the training samples into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained;

[0015] Calculate the prediction loss according to the prediction result and the answer statement corresponding to the query statement;

[0016] Counting the actual usage times of each expert model in the hybrid expert model to be trained in batch training, and calculating the expected usage frequency of each expert model;

[0017] Calculating a load balancing loss based on the actual number of times used and the expected frequency of use;

[0018] The internal parameters of the hybrid expert model to be trained are optimized according to the prediction loss and the load balancing loss to obtain a trained hybrid expert model.

[0019] In an embodiment of the present application, when training a multi-level hybrid expert model, the internal parameters are optimized in combination with the load balancing loss and the prediction loss, so that the probability of each expert model being activated in the optimized model is more balanced, thereby improving the load balancing of the expert model.

[0020] In any embodiment, the optimizing the internal parameters of the hybrid expert model to be trained according to the prediction loss and the load balancing loss comprises:

[0021] Obtain hardware balance loss and communication balance loss on different levels of hardware devices;

[0022] The internal parameters of the hybrid expert model to be trained are optimized according to the hardware balancing loss, the communication balancing loss, the prediction loss and the load balancing loss on the hardware devices at different levels.

[0023] In an embodiment of the present application, when training a multi-level hybrid expert model, the internal parameters are optimized in combination with hardware balancing loss, communication balancing loss, load balancing loss and prediction loss, so that the optimized model has improvements in terms of load balancing, computing performance, etc. of the expert model.

[0024] In any embodiment, the step of inputting the query sentence in the training sample into the hybrid expert model to be trained to obtain the prediction result output by the hybrid expert model to be trained includes:

[0025] Inputting the query sentences in the training sample into the hybrid expert model to be trained, and the compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the query sentences to obtain the first hidden layer features;

[0026] Upsampling the first hidden layer features to obtain features with the same size as the input query sentence, and deleting the first hidden layer features;

[0027] The features are input into the multi-layer hierarchical hybrid expert layer to obtain the prediction result.

[0028] In the embodiment of the present application, the first hidden layer features are not saved during the model training phase, thereby saving the storage requirements of the KV cache.

[0029] In any embodiment, the method further comprises:

[0030] receiving a query statement, and inputting the query statement into the hybrid expert model;

[0031] Performing a normalization operation on the query statement through the normalization layer to obtain a normalized query statement;

[0032] Using the encoder and decoder in the compressed multi-head attention mechanism layer to encode and decode the normalized query statement to obtain a second hidden layer feature, and obtaining a query vector, a key vector, and a value vector based on the second hidden layer feature;

[0033] Processing the query vector, the key vector, and the value vector using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain a processing result;

[0034] The processing result is processed again using the multi-layer hierarchical hybrid expert layer to obtain a query result corresponding to the query statement.

[0035] In the embodiment of the present application, the query vector, key vector and value vector are not saved in the inference stage, but the query vector, key vector and value vector are restored through the second hidden layer features, thereby saving the storage requirements of the KV cache.

[0036] In any embodiment, the normalization layer includes a first normalization module and a second normalization module;

[0037] The first normalization module is arranged before the compressed multi-head attention mechanism layer;

[0038] The second normalization module is arranged after the compressed multi-head attention mechanism layer.

[0039] In any embodiment, each level of the hybrid expert architecture includes a shared expert module and a non-shared expert module;

[0040] The number of levels of the hierarchical hybrid expert layer, and the total number of shared expert modules and non-shared expert modules are determined by the model sparsification ratio.

[0041] The embodiment of the present application determines the number of levels of the hierarchical hybrid expert layer and the total number of shared expert modules and non-shared expert modules through the model sparsification ratio, thereby reducing the complexity and computational complexity of the model.

[0042] In a second aspect, an embodiment of the present application provides a deployment device for a hybrid expert model, wherein the hybrid expert model includes multiple transformer modules, each transformer module includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-layer hierarchical hybrid expert layer;

[0043] The compressed multi-head attention mechanism layer includes an encoder-decoder, a multi-head attention mechanism module, and a position encoder;

[0044] The multi-layer hierarchical hybrid expert layer includes a multi-level hybrid expert architecture;

[0045] The deployment device comprises:

[0046] An acquisition module, used to acquire hardware devices to be deployed, wherein the hardware devices include multiple levels;

[0047] A deployment module is used to deploy the multi-level hybrid expert architecture in the corresponding levels of the hardware devices from high to low levels; wherein the level of the hybrid expert architecture deployed in the low-level hardware devices is not higher than the level of the hybrid expert architecture deployed in the high-level hardware devices.

[0048] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a bus, wherein:

[0049] The processor and the memory communicate with each other via the bus;

[0050] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0051] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium, including:

[0052] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to execute the method of the first aspect.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, wherein when the computer program instructions are read and executed by a processor, the method of the first aspect is executed.

[0054] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A schematic diagram of the architecture of a transformer module provided in an embodiment of the present application;

[0057] Figure 2 A structural diagram of a compressed multi-head attention mechanism provided in an embodiment of the present application;

[0058] Figure 3 A multi-layer hierarchical hybrid expert layer structure diagram provided in an embodiment of the present application;

[0059] Figure 4 A schematic diagram of a deployment method of a hybrid expert model provided in an embodiment of the present application;

[0060] Figure 5 A schematic diagram of the structure of a deployment device for a hybrid expert model provided in an embodiment of the present application;

[0061] Figure 6 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0064] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0065] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0067] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0068] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0069] With the widespread use of large models in online services such as search engines, chatbots, and virtual assistants, their high operating costs have become a significant obstacle. Hybrid expert models have attracted more and more attention due to their sparsity, good model performance, and efficient inference speed. However, the training of hybrid expert models brings new challenges, such as: unbalanced expert load; the requirement of lossless training; and the demand of hybrid expert models for cluster all-to-all bandwidth.

[0070] Mistral AI has successively open-sourced a large language model based on a hybrid expert model, and deepSeek has open-sourced their work on the hybrid expert model. In response to the problems in training hybrid expert models, Microsoft, Google, etc. have published their experience in MOE training, such as the overlap of communication and computing, and different types of parallel strategies. Long-context reasoning is a challenge for large models, because of the complexity of storage and computing and the context length N, which is O(N 2 ) relationship, compressing the context length is of great significance.

[0071] After long-term research, the inventor of the present application found that when deploying a hybrid expert model in a cluster, due to the existence of hierarchies on the hardware devices, the bandwidth between different servers in the cluster is different, and the bandwidth between cards in a service is also different. In addition, there are also hierarchies in the hybrid expert model, and expert models of different hierarchies have different bandwidth requirements. Therefore, in order to improve the computing performance of the hybrid expert model, when deploying the hybrid expert model, according to the hierarchy of each expert model in the hybrid expert model, the expert model is deployed on the hardware device of the corresponding hierarchy, for example: the highest-level expert model is deployed on the highest-level hardware device. Low-level expert models are deployed on relatively low-level hardware devices.

[0072] The embodiment of the present application provides a hybrid expert model, which includes multiple transformer modules, each transformer module includes a normalization layer, a compressed multi-head attention mechanism layer Compressed-Attention and a multi-layer hierarchical hybrid expert layer multi-H-MOE; Figure 1 A schematic diagram of the architecture of a transformer module provided in an embodiment of the present application is shown in FIG. Figure 1As shown. The normalized norm layer can be a normalization method such as RMS, lay norm, etc., and the norm layer position can include methods such as Pre-Norm and Post-Norm. Among them, Pre-Norm refers to performing a normalization operation first and then performing a function mapping. Post-Norm refers to performing a function mapping first and then performing a normalization operation. In an embodiment of the present application, Pre-Norm can be set before the compressed multi-head attention mechanism layer, and Post-Norm can be set after the compressed multi-head attention mechanism layer.

[0073] The hybrid expert model proposed in the embodiment of the present application can change parameters such as the number of stacked transformer modules and the hidden dimension inside the transformer module to achieve a scaling law.

[0074] The compressed multi-head attention mechanism Compressed-Attention layer includes encoder-decoder encoder-decoder, multi-head attention mechanism module multi-head-attention and position encoder ROPE; Figure 2 A compressed multi-head attention mechanism structure diagram provided in an embodiment of the present application is as follows: Figure 2 As shown:

[0075] The encoder-decoder is used to encode and decode the input tokens.

[0076] The multi-head-attention module aims to capture information from different angles and levels in the input sequence by processing multiple independent attention heads in parallel. It allows the model to abstract the sequence from multiple different representation spaces, thereby improving the model's expressiveness and learning ability. Its working principle is: for the input sequence, first generate multiple query, key, and value matrices through linear transformation. Each head has its own independent linear transformation matrix. Each head independently performs the attention mechanism, calculates the dot product of Query and Key (or other similarity metrics), and then obtains the attention weights through the softmax function, and uses these weights to weighted sum the Value. The outputs of all heads are fused by concatenation or averaging to form a unified representation. The fused representation is linearly transformed again to obtain the final output of the multi-head-attention module.

[0077] The position encoder ROPE can add the position information of tokens. In a large language model, a sentence composed of the same words in different orders may have completely different meanings. Therefore, after adding the position information, the hybrid expert model can understand the meaning of the input sentence more accurately.

[0078] Figure 3 A multi-layer hierarchical hybrid expert layer structure diagram provided in the embodiment of the present application, such as Figure 3 As shown, the multi-layer hierarchical hybrid expert layer includes a multi-level hybrid expert MOE architecture. Figure 3 What is shown in the figure are 3 levels. In practical applications, the number of layers can be reduced or increased according to actual needs, and the embodiments of the present application do not specifically limit this. The MOE architectures at different levels are isomorphic, and each level of the MOE architecture includes a non-shared expert model and may also include a shared expert model. In addition, the shared expert model and the non-shared expert model are composed of the top-K expert network of the router. That is, according to the characteristics of the input data, the probability or weight of each expert being selected is calculated, and according to the output of the routing function, the K experts with the highest probability or weight are selected, and the input data or its characteristics are divided into K parts, and each part is sent to a selected expert.

[0079] Among them, shared expert model: In the shared expert model, there is a part of experts (ie, neural network modules) that are shared, and these experts are activated when processing all inputs. These shared experts are responsible for capturing and integrating common knowledge or general information in different contexts. By sharing experts, the knowledge redundancy between the parameters of independent routing experts (ie, non-shared experts) can be reduced, and parameter efficiency can be improved. Since some experts are shared, there is no need to recalculate the outputs of these shared experts when processing new inputs, thereby reducing the computational cost.

[0080] Non-shared expert model: In the non-shared expert model, all experts are independent and they are selectively activated according to different inputs. This design allows each expert to focus on processing a specific input or task. Since each expert is independent, different experts can be flexibly combined and activated to process different inputs or tasks as needed. Moreover, each non-shared expert can focus on a specific field or task, thereby showing higher performance in that field or task.

[0081] Figure 4 A flow chart of a deployment method of a hybrid expert model provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes:

[0082] Step 401: Acquire hardware devices to be deployed, where the hardware devices include multiple levels;

[0083] Step 402: deploy the multi-level hybrid expert architecture in the corresponding levels of the hardware device from high to low levels; wherein the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

[0084] In the specific implementation process, before deploying the trained hybrid expert model to the cluster, it is necessary to know on which hardware devices in the cluster the hybrid expert model will be deployed, that is, to obtain the hardware devices to be deployed. The hardware devices are also hierarchical. Specifically, the cluster is composed of multiple servers. The servers communicate through switches. Each server contains multiple GPUs. Each GPU includes multiple cards. Therefore, the levels can be divided from high to low: switch-server-GPU. The hybrid expert model provided in the embodiment of the present application adopts a three-level MOE architecture. Each level of the MOE architecture includes 2 shared experts and 16 non-shared experts, and is composed of top-4 expert networks. There are about 16 3 =4096 experts, each token has (2+4) 3 =216 experts participate in the calculation, and the sparsity ratio is about 5%. The multi-level MOE architecture can adapt to the cluster interconnection level to achieve efficient use of cluster bandwidth, for example: all parameters in the first-level multi-H-MOE are on nodes at the same level, all parameters in the second-level multi-H-MOE are on servers at the same level, and all parameters in the third-level multi-H-MOE are on GPUs at the same level, etc.; thereby optimizing the use of hardware bandwidth at different levels in the cluster. As long as the hardware or system has a hierarchical structure, multi-H-MOEs of different levels can be deployed; even the hierarchical experts in the multi-H-MOE proposed in the present invention can be deployed on different stream processing units (SMs) in the GPU.

[0085] Therefore, the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

[0086] The embodiment of the present application optimizes the use of hardware bandwidth at different levels in the cluster by performing corresponding deployment based on the level of hardware devices and the level of the hybrid expert architecture, thereby improving the computing performance of the hybrid expert model.

[0087] Based on the above embodiment, the method further includes:

[0088] Acquire a training sample, wherein the training sample includes a question-answer pair consisting of a query sentence and an answer sentence;

[0089] Inputting the query sentences in the training samples into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained;

[0090] Calculate the prediction loss according to the prediction result and the answer statement corresponding to the query statement;

[0091] Counting the actual usage times of each expert model in the hybrid expert model to be trained in batch training, and calculating the expected usage frequency of each expert model;

[0092] Calculating a load balancing loss based on the actual number of times used and the expected frequency of use;

[0093] The internal parameters of the hybrid expert model to be trained are optimized according to the prediction loss and the load balancing loss to obtain a trained hybrid expert model.

[0094] In the specific implementation process, the hybrid expert model can be pre-trained before being deployed, and the application scenario corresponding to the hybrid expert model protected by the embodiment of the present application is described as a question-answering model.

[0095] First, a large number of training samples are obtained, each of which includes a question sentence and an answer sentence, wherein the question sentence and the answer sentence constitute a question-answer pair. The large number of training samples are divided into multiple parts, and then the hybrid expert model to be trained is batch trained.

[0096] The query statements in the training samples are input into the hybrid expert model to be trained, and the hybrid expert model to be trained analyzes and processes the query statements and outputs prediction results.

[0097] The prediction loss is calculated based on the prediction result and the answer statement corresponding to the query statement.

[0098] The actual usage times of each expert model in the batch training of the hybrid expert model to be trained are counted, and the expected usage frequency of each expert model is calculated. The expert model includes a shared expert model and a non-shared expert model. The expected usage frequency of each expert model is calculated based on an ideal uniform distribution.

[0099] The gap between the actual usage frequency and the target distribution is taken as the load balancing loss. KL divergence (Kullback-Leibler divergence) is usually used to measure this gap:

[0100] L balance =D KL (P actual ||P target )

[0101] Among them, P actual is the actual frequency of use, P target is the target usage frequency, i.e., the uniformly distributed frequency.

[0102] The load balancing loss and the prediction loss during model training are combined to form the total loss, and the internal parameters of the hybrid expert model to be trained are optimized based on the total loss to complete the training of a batch.

[0103] When the training stop condition is met, the hybrid expert model obtained from the last batch of training is used as the trained hybrid expert model. The training stop condition may be that the training batch reaches a preset batch, or the change rate of the training loss value is less than a preset value.

[0104] In an embodiment of the present application, when training a multi-level hybrid expert model, the internal parameters are optimized in combination with the load balancing loss and the prediction loss, so that the probability of each expert model being activated in the optimized model is more balanced, thereby improving the load balancing of the expert model.

[0105] Based on the above embodiment, the optimizing the internal parameters of the hybrid expert model to be trained according to the prediction loss and the load balancing loss includes:

[0106] Obtain hardware balance loss and communication balance loss on different levels of hardware devices;

[0107] The internal parameters of the hybrid expert model to be trained are optimized according to the hardware balancing loss, the communication balancing loss, the prediction loss and the load balancing loss on the hardware devices at different levels.

[0108] In the specific implementation process, in order to ensure the balance of training load. Multi-level MOE adds the balance of different hardware (nodes or different computing cards) and communication between different hardware during the training process. This avoids excessive use of a certain hardware or communication, which affects the overall efficiency. Therefore, hardware balance loss and communication balance loss are added on the basis of the model loss function. Among them, hardware balance loss and communication balance loss can also be defined by using KL divergence to measure the distance between the actual hardware and communication usage and the corresponding average usage frequency distribution.

[0109] The sum of the hardware balancing loss on hardware devices at different levels, the communication balancing loss, the prediction loss and the load balancing loss is taken as the total loss, and the internal parameters of the hybrid expert model to be trained are optimized based on the total loss to complete a batch of training.

[0110] The embodiment of the present application adds the expert load balancing loss of different levels of MOE, the load balancing loss of different computing devices, the hardware balancing loss of different computing nodes, and the communication balancing loss to the loss of traditional large model tasks during the training process. The hierarchical token-dropping strategy, such as the token-dropping strategy on different nodes and the token-dropping strategy on different servers, achieves smaller token-dropping loss and balanced loss at different levels, thereby improving the accuracy of model training.

[0111] On the basis of the above embodiment, the step of inputting the query sentence in the training sample into the hybrid expert model to be trained to obtain the prediction result output by the hybrid expert model to be trained includes:

[0112] Inputting the query sentences in the training sample into the hybrid expert model to be trained, and the compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the query sentences to obtain the first hidden layer features;

[0113] Upsampling the first hidden layer features to obtain features with the same size as the input query sentence, and deleting the first hidden layer features;

[0114] The features are input into the multi-layer hierarchical hybrid expert layer to obtain the prediction result.

[0115] In the specific implementation process, when the hybrid expert model to be trained is trained, the compressed multi-head attention mechanism layer in the hybrid expert model to be trained processes the received input data as follows:

[0116] After the query sentence of the training sample is input into the hybrid expert model to be trained, if the first layer in the hybrid expert model to be trained is a normalization layer, the query sentence is normalized by the normalization layer. If the first layer is a compressed multi-head attention mechanism layer, the compressed multi-head attention mechanism layer processes the query sentence.

[0117] Therefore, the input data in the embodiment of the present application refers to the data input into the compressed multi-head attention mechanism layer, which may be a query statement or a normalized result.

[0118] The Compressed-Attention layer uses an encoder-decoder based encoder to encode and decode the input tokens. The Compressed-Attention layer adds a codec before the multi-head-attention module in the original transformer. The codec first downsamples to form pyramid hidden features of different levels (i.e., the first hidden feature Latent), such as: the length of the input first hidden feature is 5210, and the lengths of the first hidden features generated after three downsamplings are 256, 128, and 64 respectively. The three pyramid downsampled first hidden features are combined to form 256+128+64=448 layers of hidden feature vectors; then the feature vector length of the codec is restored to 5210 by upsampling. This encoding and decoding process uses the Low-Rank method to reduce the number of weight parameters. The number of codec layers and the dimension of the first hidden feature are determined by the model's expected KV cache compression ratio. The embodiment of the present application uses low-rank compression: the KV cache is compressed into a low-rank latent vector through the coding layer, thereby reducing storage requirements. In the inference phase, this compression significantly reduces the access to the KV cache and speeds up processing. Assuming that the dimension of the original hidden layer feature is d, the dimension of the encoded and decoded hidden layer feature is r, the compression ratio of the KV cache is compression, which is specifically expressed as follows:

[0119]

[0120] The encoder-decoder encoder encodes and decodes the input tokens; implements the latent representation of the input features and compresses the KV cache storage requirements. During the model training phase, the Compressed-Attention layer does not need to save the first latent layer features (Latent), that is, the first latent layer features are deleted.

[0121] The features obtained after upsampling are input into the multi-layer hierarchical hybrid expert layer, which analyzes them to obtain the prediction results.

[0122] In the embodiment of the present application, the first hidden layer features are not saved during the model training phase, thereby saving the storage requirements of the KV cache.

[0123] Based on the above embodiment, the method further includes:

[0124] receiving a query statement, and inputting the query statement into the hybrid expert model;

[0125] Performing a normalization operation on the query statement through the normalization layer to obtain a normalized query statement;

[0126] Using the encoder and decoder in the compressed multi-head attention mechanism layer to encode and decode the normalized query statement to obtain a second hidden layer feature, and obtaining a query vector, a key vector, and a value vector based on the second hidden layer feature;

[0127] Processing the query vector, the key vector, and the value vector using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain a processing result;

[0128] The processing result is processed again using the multi-layer hierarchical hybrid expert layer to obtain a query result corresponding to the query statement.

[0129] In the specific implementation process, when the trained hybrid expert model is used for reasoning, a query statement is received and input into the hybrid expert model. The query statement is first normalized by the normalization layer in the hybrid expert model to obtain the normalized query statement. Then the normalized query statement is input into the compressed multi-head attention mechanism layer. The encoder and decoder in the compressed multi-head attention mechanism layer encode and decode the normalized query statement to obtain the second hidden layer features, and the query vector (query), key vector (key) and value vector (value) are obtained based on the second hidden layer features. The multi-head attention mechanism module processes the query vector, key vector and value vector to obtain the processing result. Each head independently executes the attention mechanism, calculates the dot product (or other similarity measure) of the query and the key, and then obtains the attention weight through the softmax function, and uses these weights to perform weighted summation on the value. The outputs of all heads are fused by concatenation or averaging to form a unified representation. The fused representation is linearly transformed again to obtain the final output of Multi-Head Attention. The final output result of Multi-HeadAttention is then input into the multi-layer hierarchical hybrid expert layer. Based on the input result, the multi-layer hierarchical hybrid expert layer matches k expert models for it and activates these k expert models. These expert models process the input result to obtain the final query result.

[0130] It should be noted that, since a hybrid expert model includes multiple transformer modules, the input of the second transformer module is the output of the first transformer module, and so on, until the last transformer module outputs the final query result.

[0131] In the embodiment of the present application, during the model inference stage, the Compressed-Attention layer needs to save the latent features (Latent) of the latent layer QKV instead of the KV cache, thereby saving the KV cache storage requirement.

[0132] Based on the above embodiment, each level of the hybrid expert architecture includes a shared expert module and a non-shared expert module;

[0133] The number of levels of the hierarchical hybrid expert layer, and the total number of shared expert modules and non-shared expert modules are determined by the model sparsification ratio.

[0134] In the specific implementation process, the sparsification ratio refers to the probability that each input sample is assigned to an expert. It can determine the complexity and computational cost of the model. A lower sparsification ratio will cause more input samples to be assigned to experts, increasing the complexity and computational cost of the model; while a higher sparsification ratio will reduce these situations. The number of experts and top_K parameters of an efficient hybrid expert model can be determined based on the sparsification ratio:

[0135] For example, the hybrid expert model of the embodiment of the present application has an L-level hybrid expert architecture, each level has s shared expert models and m non-shared expert models, and each time k non-shared expert models are selected from the m non-shared experts for activation. Then the sparsity of the hybrid expert model is:

[0136]

[0137] When the sparsity is known, the top-k parameters and the number of shared expert models and non-shared expert models are adjusted so that the sparsification ratio of the final model meets the requirements.

[0138] The embodiment of the present application stacks multiple transformer modules, which is a solution with less information redundancy, finer-grained experts, more balanced model training loss and more efficient reasoning deployment. MOE often requires all-to-all bandwidth requirements during training. The embodiment of the present application fully utilizes the interconnection bandwidth between different levels of the training cluster to improve the efficiency of MOE training.

[0139] It should be noted that the hybrid expert model provided in the embodiment of the present application can be implemented through YARN and other methods to realize long context models, and can be used in other aspects such as reinforcement learning combined with Monte Carlo tree search.

[0140] The embodiment of the present application determines the number of levels of the hierarchical hybrid expert layer and the total number of shared expert modules and non-shared expert modules through the model sparsification ratio, thereby reducing the complexity and computational complexity of the model.

[0141] Figure 5 This is a schematic diagram of a deployment device structure of a hybrid expert model provided in an embodiment of the present application. The device may be a module, program segment or code on an electronic device. It should be understood that the device is similar to the above-mentioned Figure 4 The method embodiment corresponds to and can be executed Figure 4 The various steps involved in the method embodiment and the specific functions of the device can be found in the description above. To avoid repetition, detailed description is appropriately omitted here. The hybrid expert model includes multiple transformer modules, each of which includes a normalization layer, a compressed multi-head attention mechanism layer and a multi-layer hierarchical hybrid expert layer;

[0142] The compressed multi-head attention mechanism layer includes an encoder-decoder, a multi-head attention mechanism module, and a position encoder;

[0143] The multi-layer hierarchical hybrid expert layer includes a multi-level hybrid expert architecture;

[0144] The deployment device comprises:

[0145] The acquisition module 501 is used to acquire hardware devices to be deployed, and the hardware devices include multiple levels;

[0146] The deployment module 502 is used to deploy the multi-level hybrid expert architecture in the corresponding level of the hardware device from high to low level; wherein the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

[0147] Based on the above embodiment, the device further includes a training module, which is used to:

[0148] Acquire a training sample, wherein the training sample includes a question-answer pair consisting of a query sentence and an answer sentence;

[0149] Inputting the query sentences in the training samples into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained;

[0150] Calculate the prediction loss according to the prediction result and the answer statement corresponding to the query statement;

[0151] Counting the actual usage times of each expert model in the hybrid expert model to be trained in batch training, and calculating the expected usage frequency of each expert model;

[0152] Calculating a load balancing loss based on the actual number of times used and the expected frequency of use;

[0153] The internal parameters of the hybrid expert model to be trained are optimized according to the prediction loss and the load balancing loss to obtain a trained hybrid expert model.

[0154] Based on the above embodiment, the training module is specifically used for:

[0155] Obtain hardware balance loss and communication balance loss on different levels of hardware devices;

[0156] The internal parameters of the hybrid expert model to be trained are optimized according to the hardware balancing loss, the communication balancing loss, the prediction loss and the load balancing loss on the hardware devices at different levels.

[0157] Based on the above embodiment, the training module is specifically used for:

[0158] Inputting the query sentences in the training sample into the hybrid expert model to be trained, and the compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the input data to obtain the first hidden layer features;

[0159] Upsampling the first hidden layer features to obtain features with the same size as the input query sentence, and deleting the first hidden layer features;

[0160] The features are input into the multi-layer hierarchical hybrid expert layer to obtain the prediction result.

[0161] Based on the above embodiment, the device further includes a reasoning module, which is used to:

[0162] receiving a query statement, and inputting the query statement into the hybrid expert model;

[0163] Performing a normalization operation on the query statement through the normalization layer to obtain a normalized query statement;

[0164] Using the encoder and decoder in the compressed multi-head attention mechanism layer to encode and decode the normalized query statement to obtain a second hidden layer feature, and obtaining a query vector, a key vector, and a value vector based on the second hidden layer feature;

[0165] Processing the query vector, the key vector, and the value vector using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain a processing result;

[0166] The processing result is processed again using the multi-layer hierarchical hybrid expert layer to obtain a query result corresponding to the query statement.

[0167] On the basis of the above embodiment, the normalization layer includes a first normalization module and a second normalization module;

[0168] The first normalization module is arranged before the compressed multi-head attention mechanism layer;

[0169] The second normalization module is arranged after the compressed multi-head attention mechanism layer.

[0170] Based on the above embodiment, each level of the hybrid expert architecture includes a shared expert module and a non-shared expert module;

[0171] The number of levels of the hierarchical hybrid expert layer, and the total number of shared expert modules and non-shared expert modules are determined by the model sparsification ratio.

[0172] Figure 6 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device includes: a processor (processor) 601, a memory (memory) 602 and a bus 603; wherein,

[0173] The processor 601 and the memory 602 communicate with each other via the bus 603;

[0174] The processor 601 is used to call the program instructions in the memory 602 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a hardware device to be deployed, the hardware device including multiple levels; deploying the multi-level hybrid expert architecture in the corresponding level in the hardware device from high to low levels; wherein the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

[0175] The processor 601 may be an integrated circuit chip with signal processing capabilities. The processor 601 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. It may implement or execute various methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0176] The memory 602 may include but is not limited to random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), etc.

[0177] The present embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a hardware device to be deployed, wherein the hardware device includes multiple levels; deploying the multi-level hybrid expert architecture in the corresponding levels of the hardware device from high to low levels; wherein the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

[0178] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions enable the computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining a hardware device to be deployed, wherein the hardware device includes multiple levels; deploying the multi-level hybrid expert architecture in the corresponding levels of the hardware device from high to low levels; wherein the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

[0179] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0180] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0182] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0183] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A deployment method for a hybrid expert model, It is characterized in that The hybrid expert model includes multiple transformer modules, each of which includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-layer hierarchical hybrid expert layer; The compressed multi-head attention mechanism layer includes an encoder-decoder, a multi-head attention mechanism module, and a position encoder; The multi-layer hierarchical hybrid expert layer includes a multi-level hybrid expert architecture; The deployment method includes: Acquire hardware equipment to be deployed, wherein the hardware equipment includes multiple levels; The multi-level hybrid expert architecture is deployed in the corresponding level of the hardware device from high to low level; wherein the level of the hybrid expert architecture deployed in the low-level hardware device is not higher than the level of the hybrid expert architecture deployed in the high-level hardware device.

2. The method according to claim 1, It is characterized in that The method further comprises: Acquire a training sample, wherein the training sample includes a question-answer pair consisting of a query sentence and an answer sentence; Inputting the query sentences in the training samples into the hybrid expert model to be trained to obtain the prediction results output by the hybrid expert model to be trained; Calculate the prediction loss according to the prediction result and the answer statement corresponding to the query statement; Counting the actual usage times of each expert model in the hybrid expert model to be trained in batch training, and calculating the expected usage frequency of each expert model; Calculating a load balancing loss based on the actual number of times used and the expected frequency of use; The internal parameters of the hybrid expert model to be trained are optimized according to the prediction loss and the load balancing loss to obtain a trained hybrid expert model.

3. The method according to claim 2, It is characterized in that The optimizing the internal parameters of the hybrid expert model to be trained according to the prediction loss and the load balancing loss comprises: Obtain hardware balance loss and communication balance loss on different levels of hardware devices; The internal parameters of the hybrid expert model to be trained are optimized according to the hardware balancing loss, the communication balancing loss, the prediction loss and the load balancing loss on the hardware devices at different levels.

4. The method according to claim 2, It is characterized in that The step of inputting the query sentence in the training sample into the hybrid expert model to be trained to obtain the prediction result output by the hybrid expert model to be trained comprises: Inputting the query sentences in the training sample into the hybrid expert model to be trained, and the compressed multi-head attention mechanism layer in the hybrid expert model to be trained downsamples the input data to obtain the first hidden layer features; Upsampling the first hidden layer features to obtain features with the same size as the input query sentence, and deleting the first hidden layer features; The features are input into the multi-layer hierarchical hybrid expert layer to obtain the prediction result.

5. The method according to claim 4, It is characterized in that The method further comprises: receiving a query statement, and inputting the query statement into the hybrid expert model; Performing a normalization operation on the query statement through the normalization layer to obtain a normalized query statement; Using the encoder and decoder in the compressed multi-head attention mechanism layer to encode and decode the normalized query statement to obtain a second hidden layer feature, and obtaining a query vector, a key vector, and a value vector based on the second hidden layer feature; Processing the query vector, the key vector, and the value vector using the multi-head attention mechanism module in the compressed multi-head attention mechanism layer to obtain a processing result; The processing result is processed again using the multi-layer hierarchical hybrid expert layer to obtain a query result corresponding to the query statement.

6. The method according to claim 1, It is characterized in that The normalization layer includes a first normalization module and a second normalization module; The first normalization module is arranged before the compressed multi-head attention mechanism layer; The second normalization module is arranged after the compressed multi-head attention mechanism layer.

7. The method according to any one of claims 1 to 6, It is characterized in that Each level of the hybrid expert architecture includes shared expert modules and non-shared expert modules; The number of levels of the hierarchical hybrid expert layer, and the total number of shared expert modules and non-shared expert modules are determined by the model sparsification ratio.

8. A deployment device for a hybrid expert model, It is characterized in that The hybrid expert model includes multiple transformer modules, each of which includes a normalization layer, a compressed multi-head attention mechanism layer, and a multi-layer hierarchical hybrid expert layer; The compressed multi-head attention mechanism layer includes a codec, a multi-head attention mechanism module and a position encoder; The multi-layer hierarchical hybrid expert layer includes a multi-level hybrid expert architecture; The deployment device comprises: An acquisition module, used to acquire hardware devices to be deployed, wherein the hardware devices include multiple levels; A deployment module is used to deploy the multi-level hybrid expert architecture in the corresponding levels of the hardware devices from high to low levels; wherein the level of the hybrid expert architecture deployed in the low-level hardware devices is not higher than the level of the hybrid expert architecture deployed in the high-level hardware devices.

9. An electronic device, It is characterized in that include: processor, memory and bus, wherein, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, It is characterized in that The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 7.

11. A computer program product, It is characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images