Methods, systems, apparatus, media, and products for hybrid expert models

Abstract

The invention discloses a method, a system, a device, a medium and a product for a hybrid expert model. The hybrid expert model comprises a predictor, a gating module and a model projection layer, and the method comprises the steps that feature vectors representing lexical elements are input into the predictor, so that the predictor generates importance scores corresponding to all neurons in the model projection layer based on the feature vectors; passing the importance score through a gating module based on a modified linear unit to generate a gating value indicating whether each neuron in a model projection layer is activated or not; and controlling a model projection layer to perform sparse projection calculation on the feature vector based on the gating value, and generating a projection calculation result corresponding to the feature vector.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large language models, and in particular to a method, system, device, medium and product for a mixture of expert models. BACKGROUND

[0002] The mixture of experts (MoE) combined with the activation sparsity technique has become an important direction for accelerating large language models. Existing activation sparsity techniques use a fixed selection strategy, and all input tokens use the same sparsity regardless of their complexity. This results in unnecessary computational resources being wasted for simple tokens, and complex tokens may lose accuracy due to insufficient activated neurons. SUMMARY

[0003] The method for a mixture of expert models according to an embodiment of the present application, the mixture of expert models comprising a predictor, a gating module and a model projection layer, the method comprising: inputting a feature vector representing a token into the predictor to cause the predictor to generate an importance score for each neuron in the corresponding model projection layer based on the feature vector; passing the importance score through the gating module based on a rectified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated; and based on the gating value, controlling the model projection layer to perform sparse projection calculation on the feature vector to generate a projection calculation result corresponding to the feature vector.

[0004] The system for a mixture of expert models according to an embodiment of the present application, the mixture of expert models comprising a predictor, a gating module and a model projection layer, the system comprising: an importance score unit configured to input a feature vector representing a token into the predictor to cause the predictor to generate an importance score for each neuron in the corresponding model projection layer based on the feature vector; a gating value unit configured to pass the importance score through the gating module based on a rectified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated; and a projection calculation unit configured to, based on the gating value, control the model projection layer to perform sparse projection calculation on the feature vector to generate a projection calculation result corresponding to the feature vector.

[0005] The device for a mixture of expert models according to an embodiment of the present application comprises: a processor; and a memory having computer executable instructions stored thereon, wherein the computer executable instructions, when executed by the processor, cause the processor to perform the above method.

[0006] The computer readable storage medium according to an embodiment of the present application has computer executable instructions stored thereon, wherein the computer executable instructions, when executed by the processor, cause the processor to perform the above method.

[0007] The computer program product according to the embodiments of the present application comprises computer executable instructions, wherein the computer executable instructions, when executed by a processor, cause the processor to perform the above method. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present application can be better understood with reference to the following description of embodiments of the present application taken in conjunction with the accompanying drawings, in which: Figure 1 A structural schematic diagram of a hybrid expert model according to an embodiment of the present application is shown.

[0009] Figure 2 A flowchart of a method for a hybrid expert model according to an embodiment of the present application is shown.

[0010] Figure 3 A schematic block diagram of a system for a hybrid expert model according to an embodiment of the present application is shown.

[0011] Figure 4 A schematic diagram of a computer system that can implement a method and apparatus for a hybrid expert model according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0012] Features and exemplary embodiments of various aspects of the present application will be described below in detail. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without some or all of these specific details. The description of the embodiments is merely intended to provide a better understanding of the present application by showing examples of the present application. The present application is in no way limited to any specific configuration and algorithm set forth below, but covers any modification, replacement, and improvement of elements, components, and algorithms without departing from the spirit of the present application. In the accompanying drawings and the following description, well-known structures and techniques are not shown in order to avoid unnecessary obscuring of the present application.

[0013] Generally, a large language model comprises tens or even hundreds of Transformer layers. In a large language model of MoE architecture, a standard Feed Forward Networks (FFN) sublayer of part of the Transformer layers is replaced by a hybrid expert layer. Each hybrid expert layer comprises a router and a plurality of hybrid expert models.

[0014] A token is the smallest text unit of model input. A plurality of tokens can be obtained by processing a sentence by a tokenizer. After processing by at least one of an embedding layer and a Transformer layer in a large language model, a high-dimensional, dense feature vector can be obtained. The feature vector contains semantic, grammatical information of the token and its relationship with the context, and can represent the original token.

[0015] When the feature vector corresponding to a word piece is sent to a hybrid expert model, the feature vector will be subjected to matrix multiplication operation with the weight matrix of the model projection layer in the hybrid expert model, and the calculation result will be taken as the output of the hybrid expert model. This matrix multiplication operation occupies the main calculation overhead in model inference. In order to improve the operation efficiency while maintaining the model performance, a combined solution of hybrid expert model and activation sparsity technology is proposed, that is, the neurons participating in the calculation in the model are subjected to sparse control, that is, only part of the neurons in the model are activated and participate in the calculation, so as to speed up the model inference.

[0016] The existing activation sparsity technology, such as the TopK selection strategy, adopts a fixed selection strategy, and all input word pieces use the same sparsity (for example, 25% of the neurons are reserved) regardless of the complexity. This results in that simple word pieces waste unnecessary calculation resources in the model, and complex word pieces may lose accuracy due to insufficient number of activated neurons.

[0017] According to the hybrid expert model of the embodiment of the present application, a predictor and a rectified linear unit (ReLU) based gating module are arranged. According to the embodiment of the present application, when the feature vector of the indicator word piece is input into the predictor, the predictor outputs an importance score, and the importance score is subjected to gating value through the rectified linear unit based gating module, so as to be used for sparse projection calculation of the model projection layer. The predictor that has completed training will dynamically output corresponding importance scores according to different feature vectors, and the gating value obtained through the gating module can activate different numbers of neurons, so that the activation sparsity of the model projection layer in the hybrid expert model dynamically changes according to the input feature vector, overcoming the defects of the original fixed sparsity. The rectified linear unit based gating module has the advantages of continuity, full-differentiability and unbiased gradient transmission, thereby providing differentiable control for the application of the hybrid expert model.

[0018] Figure 1 A structural schematic diagram of a hybrid expert model 100 according to an embodiment of the present application is shown. The hybrid expert model 100 includes a predictor, a gating module and a model projection layer. The model projection layer can include an upward projection layer, a gating projection layer and a downward projection layer.

[0019] Figure 2 A flowchart of a dynamic control method 200 of the model internal sparsity according to an embodiment of the present application is shown. As shown in FIG. 2, the method 200 includes the following steps. Figure 2As shown, the dynamic control method 200 includes steps S201-S203. S201: inputting a feature vector representing a word into a predictor to make the predictor generate importance scores of each neuron in the corresponding model projection layer based on the feature vector. S202: passing the importance scores through a gated module based on a rectified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated. S203: based on the gating value, controlling the model projection layer to perform sparse projection calculation on the feature vector to generate a projection calculation result corresponding to the feature vector.

[0020] In the method according to the embodiment of the application, the predictor can be a low-rank predictor network, which can be represented as: ; wherein , , is the dimension of the input feature vector, r and d correspond to different dimensions respectively, and , so that the predictor behaves as a low-rank dimension.

[0021] As shown in Figure 1 , assuming a feature vector x, inputting the predictor obtains importance scores of each neuron in the corresponding model projection layer, which can be represented as . The importance score corresponding to each neuron represents the original prediction of the predictor on the contribution degree of the corresponding neuron to the model output under the current input. Passing the importance scores through a gated module based on a rectified linear unit generates a gating value, which can be represented as . Here, each value in the gating value indicates whether the corresponding neuron is activated: when the corresponding value is equal to 0, it indicates that the corresponding neuron is not activated; when the corresponding value is greater than 0, it indicates that the corresponding neuron is activated. Specifically, the overall situation of whether all neurons are activated can be described by the instantaneous activation sparsity of the current model projection layer, which is the ratio of the number of currently unactivated neurons to the number of all neurons. The trained predictor will dynamically output the corresponding importance scores according to different feature vectors x, and the gating value obtained by the gating module can activate different neurons, so the activation sparsity of the model projection layer dynamically changes according to the input feature vector, overcoming the defects of the original fixed sparsity.

[0022] As shown in Figure 1As shown, in some embodiments, controlling the projection layer of the control model to perform sparse projection calculation on the feature vector to generate the projection calculation result of the corresponding feature vector may include: obtaining the first projection result and the second projection result obtained after the feature vector passes through the upward projection layer and the gated projection layer respectively; multiplying the gate value with the first projection result and the second projection result element by element to obtain the first multiplication result and the second multiplication result; multiplying the first multiplication result and the second multiplication result element by element to obtain the intermediate feature; and controlling the downward projection layer to selectively calculate the intermediate feature based on the gate value to generate the projection calculation result. The second projection result can also be processed by an activation function (e.g., a Sigmoid linear unit, SiLU) before performing element-wise multiplication. When the feature vector passes through the upward projection layer or the gated projection layer, it actually performs matrix multiplication with the weight matrix of the upward projection layer or the gated projection layer, and the result is the first projection result or the second projection result. Here, controlling the downward projection layer to perform selective calculation means only calculating the rows where the corresponding value in the gate value is greater than 0. The calculation specifically involves matrix multiplication of the weight matrix of the downward projection layer and the intermediate feature. Assume the weight matrix of the downward projection layer is: , intermediate features are If the gate value g = [0, 0.3, 0, 0.8] at this time, then the intermediate features... and The value is 0. In this case, selective computation only involves the activation portion with a gating value greater than 0. The calculation process is as follows: ; .

[0023] In the method according to an embodiment of the present invention, the hybrid expert model can be deployed directly through inference, that is, by directly using the dynamic gating value generated by the gating module based on the modified linear unit. By default, neurons with a gating value greater than 0 are considered to be activated, so that the expert model maintains dynamic sparsity characteristics and is suitable for scenarios that are not sensitive to latency.

[0024] In the method according to embodiments of the present invention, the hybrid expert model can also be deployed for inference with a fixed target threshold. Specifically, in some embodiments, controlling the model projection layer to perform sparse projection calculation on the feature vector may include: considering neurons in the model projection layer with a gating value greater than the target threshold as activated, and controlling the neurons considered as activated in the model projection layer to perform sparse projection calculation on the feature vector. Here, the target threshold is greater than 0. Compared to the default activation condition where the gating value of the neuron is greater than 0 when directly deploying the hybrid expert model for inference, the comparison between the target threshold and the gating value further increases the threshold for neuron activation. This deployment scheme is suitable for hardware deployments requiring a fixed amount of computation.

[0025] In the method according to the embodiment of the application, the hybrid expert model can be obtained by joint training, and each training step in the joint training can include: obtaining a current training output result of forward propagation of a training sample in a to-be-trained model, determining a current total loss based on the current training output result according to a first loss formula, and performing reverse update on the to-be-trained model by using the current total loss.

[0026] The to-be-trained model includes at least one transformer layer, and the at least one transformer layer implements feedforward calculation by using a hybrid expert model.

[0027] The first loss formula includes: . is the current total loss. is a current language model loss corresponding to the current training output result, and the current language model loss can be determined based on the current training output result by using a method known in the field of large language models, which will not be described in detail herein. is a current regularization loss about activation sparsity corresponding to the current training output result. The activation sparsity is a ratio of a number of neurons that are not activated to a total number of all neurons in the to-be-trained model. Since direct training of the gating module often leads to a lower sparsity (the model tends to activate more neurons to increase the capacity), the regularization loss is introduced in this embodiment to force a higher sparsity. λ is a current adaptive coefficient about the activation sparsity corresponding to the current training output result, and λ is a positive value.

[0028] In some embodiments, the current adaptive coefficient can be determined by: determining a current average sparsity based on the current training output result; increasing an adaptive coefficient of the last training to obtain the current adaptive coefficient in a case where the current average sparsity is lower than a target sparsity; and reducing the adaptive coefficient of the last training to obtain the current adaptive coefficient in a case where the current average sparsity is higher than the target sparsity.

[0029] In some embodiments, determining the current average sparsity can include: determining the current average sparsity based on the current training output result according to an average sparsity formula, and the average sparsity formula includes: , wherein S is the current average sparsity, L is a number of transformer layers in the to-be-trained model, T is a sequence length of tokens corresponding to the training sample, d is a number of all neurons in each hybrid expert model, is a gating value of an nth neuron corresponding to an mth transformer layer and an tth token in the current training output result, is a gating value of an nth neuron corresponding to an mth transformer layer and an tth token in the current training output result, is a condition that An indicator function that takes 1 if the value is true and 0 otherwise.

[0030] In some embodiments, increasing the adaptive coefficient from the previous training iteration to obtain the current adaptive coefficient may include multiplying the adaptive coefficient from the previous training iteration by a preset multiplier, where the preset multiplier is greater than 1. Correspondingly, decreasing the adaptive coefficient from the previous training iteration to obtain the current adaptive coefficient may include multiplying the adaptive coefficient from the previous training iteration by the reciprocal of the preset multiplier.

[0031] Therefore, assuming this training is the i-th iteration, the calculation of the current adaptive coefficients can be expressed as: .in, These are the current adaptive coefficients. These are the adaptive coefficients from the previous training iteration, and their initial values. It can be set to a very small positive number, for example 1-k / d is the target sparsity, which can be set to 75% for example. The preset multiplier is greater than 1, for example, it can be 1.2. This is a multiplier determined based on the relationship between the target sparsity and the current average sparsity, multiplied by the adaptive coefficients from the previous training iteration. Specifically, if the current average sparsity is lower than the target sparsity, the multiplier is... To increase the adaptive coefficient; if the current average sparsity is higher than the target sparsity, the multiplier is... The reciprocal of is used to reduce the adaptive coefficient.

[0032] In the method according to an embodiment of the present invention, the adaptive coefficients of the previous training are adjusted based on the relationship between the target sparsity rate and the current average sparsity rate to obtain the current adaptive coefficients. Other adjustment methods can also be used, such as using different calculation methods or different values, as long as the current adaptive coefficients meet the adjustment target.

[0033] In some embodiments, the current regularization loss can be determined by: determining the current regularization loss based on the current training output according to a second loss formula, wherein the second loss formula includes: L represents the number of converter layers in the model to be trained, T represents the sequence length of the corresponding word units in the training samples, and d represents the total number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, and the gating value corresponding to the t-th word in the n-th neuron.

[0034] Here, the second loss formula is actually the L1 regularization loss, which can be expressed as: .

[0035] Due to the non-negative output property of the rectified linear unit, the L1 norm is equal to the direct summation, so the second loss formula above can be obtained.

[0036] In some embodiments, in order to achieve load balancing between neurons, the calculation of the regularization loss can introduce an average activation ratio weight, and the current regularization loss can be determined by determining the current regularization loss based on the current training output result according to a third loss formula, the third loss formula comprising: , represents the current regularization loss considering load balancing, L is the number of transformer layers in the model to be trained, T is the sequence length of the word pieces corresponding to the training sample, d is the number of all neurons in each hybrid expert model, is the current regularization loss considering load balancing, L is the number of transformer layers in the model to be trained, T is the sequence length of the word pieces corresponding to the training sample, d is the number of all neurons in each hybrid expert model, is the gating value corresponding to the nth neuron of the transformer layer of the (l-1)th word piece in the current training output result, is the average activation ratio weight of the nth neuron of the transformer layer of the (l-1)th word piece determined based on the current training output result.

[0037] In some embodiments, the average activation ratio weight can be determined by determining the average activation ratio weight based on the current training output result according to an average activation ratio weight formula, the average activation ratio weight formula comprising: , k is the number of neurons expected to be activated in the hybrid expert model. Here, the weight serves as a coefficient for the corresponding gating value, modifying the gradient of the non-zero output to This mechanism will punish neurons that receive relatively more word pieces by driving their gating values to zero faster, achieving load balancing. Unlike TopK routing, the gating values output by the rectified linear unit-based gating module according to the method of the present embodiment can be arbitrarily small, so that tends to 0. Therefore, the adaptive coefficient λ in the present embodiment is not a fixed value, so as to avoid the situation that the routing collapses to 0. Thanks to the adaptive update of the adaptive coefficient λ, the sparse rate control and load balancing can be balanced in a single formula.

[0038] ​According to the description of each training step in the joint training, the whole process of the joint training can be divided into three stages of a warm-up period, a sparsification period, and a stable period according to performance. In the warm-up period, the adaptive coefficient λ remains a small initial value, the model is initialized for training in a way that the feedforward network is approximately dense, the predictor preliminarily learns the feature representation of the input; after entering the sparsification period, λ starts to adaptively increase according to the deviation of the current average sparsity from the target sparsity, driving the average sparsity of the model to gradually approach the preset target sparsity, and the predictor learns to predict the optimal sparse mode corresponding to different inputs; finally, in the stable period, λ converges to a dynamic equilibrium point that can keep the average sparsity stably maintained near the target sparsity, and the model continues to optimize under this sparse constraint, realizing efficient and stable convergence. The three stages ensure the smoothness and effectiveness of the collaborative evolution of the predictor and the model projection layer under the sparse constraint.

[0039] The method according to the embodiment of the application realizes multi-level dynamic calculation allocation. Firstly, word-level dynamics are realized according to the complexity of different words to activate different numbers of neurons. For simple words, fewer neurons are activated to save calculation; for complex words, more neurons are activated to ensure accuracy. Then, model-level dynamics are realized, and different layers can learn different sparse modes. The shallow layer may extract basic features with denser calculation, and the deep layer may be sparser and perform high-level semantic processing. Finally, expert-level dynamics are realized, and different experts can have different sparse characteristics, for example, general experts correspond to a lower sparsity and process common patterns, and professional experts correspond to a higher sparsity and process specific fields.

[0040] Compared with the prior art, the method according to the embodiment of the application has achieved significant improvement in multiple dimensions.

[0041] In terms of training efficiency, the nondifferentiability of the original TopK operation leads to training difficulties. Although the Balanced TopK module after the predictor output alleviates the neuron degradation problem, TopK itself is still a discrete and nondifferentiable operation. When backpropagation, the gradient can only be transmitted through the Straight-Through Estimator (STE) approximation, resulting in a deviation in gradient estimation, making it difficult for the predictor to learn the optimal sparse mode, slowing down the training convergence speed, and limiting the final performance. According to the embodiment of the application, the fully differentiable design of the hybrid expert model ensures accurate gradient transmission, enabling the predictor to learn the optimal sparse mode faster, with a convergence speed improvement of about 20-30%.

[0042] Regarding model accuracy, existing technologies lack dynamic computational allocation at the hierarchical and token levels. In existing schemes, the sparsity rate of each layer and each token is preset and fixed, making it impossible to dynamically adjust the allocation of computational resources according to the complexity of the actual input. According to embodiments of the present invention, the dynamic sparsity mechanism allows complex tokens to call more computational resources, effectively avoiding the information loss caused by fixed sparsity, and reducing model perplexity by 5-10% at the same sparsity rate.

[0043] Regarding computational resource utilization, the method according to embodiments of the present invention can automatically allocate computational load based on input complexity, enabling highly sparse computation for simple terms, further reducing the overall computational load by 10-20%, while maintaining or even improving model accuracy. In terms of deployment and application, the method according to embodiments of the present invention provides two inference modes: dynamic sparsity and fixed-threshold sparsity. The optimal deployment strategy can be flexibly selected based on actual hardware constraints and latency requirements, significantly enhancing the system's practicality and adaptability.

[0044] Figure 3 A schematic block diagram of a system 300 for a hybrid expert model according to an embodiment of the present invention is shown. Figure 3 As shown, the system 300 for a hybrid expert model includes an importance score unit 301, a gating value unit 302, and a projection calculation unit 303. The importance score unit 301 is configured to input feature vectors representing words into a predictor, enabling the predictor to generate importance scores for each neuron in the model's projection layer based on the feature vectors. The gating value unit 302 is configured to pass the importance scores through a gating module based on a modified linear unit to generate gating values ​​indicating whether each neuron in the model's projection layer is activated. The projection calculation unit 303 is configured to control the model's projection layer to perform sparse projection calculations on the feature vectors based on the gating values, generating projection calculation results for the corresponding feature vectors.

[0045] In some embodiments, the hybrid expert model is obtained through joint training. Each training step in the joint training may include: obtaining the current training output result of the training samples propagating forward in the model to be trained, wherein the model to be trained includes at least one transformer layer, and the at least one transformer layer implements its feedforward calculation through a hybrid expert model; determining the current total loss based on the current training output result according to a first loss formula, wherein the first loss formula is as follows: , This represents the current total loss. The loss of the current language model corresponds to the current training output. a current regularization loss corresponding to the current training output result about an activation sparsity, the activation sparsity being a ratio of a number of neurons that are not activated in the to-be-trained model to a total number of all neurons, and the current adaptive coefficient corresponding to the current training output result about the activation sparsity being positive; and performing a backward update on the to-be-trained model by using the current total loss.

[0046] In some embodiments, the current adaptive coefficient can be determined by: determining a current average sparsity based on the current training output result; in a case that the current average sparsity is lower than a target sparsity, increasing the adaptive coefficient of the last training to obtain the current adaptive coefficient; and in a case that the current average sparsity is higher than the target sparsity, decreasing the adaptive coefficient of the last training to obtain the current adaptive coefficient.

[0047] In some embodiments, the determination of the current average sparsity can include: determining the current average sparsity based on the current training output result according to an average sparsity formula, the average sparsity formula including: wherein S is the current average sparsity, L is a number of the transformer layers in the to-be-trained model, T is a sequence length of the tokens corresponding to the training sample, and d is a number of all neurons in each mixed expert model. is a gating value corresponding to the nth neuron of the transformer layer of the lth layer and the tth token in the current training output result. is a gating value corresponding to the nth neuron of the transformer layer of the lth layer and the tth token in the current training output result.

[0048] In some embodiments, the increasing of the adaptive coefficient of the last training to obtain the current adaptive coefficient can include: multiplying the adaptive coefficient of the last training by a preset multiplier, the preset multiplier being greater than 1; and the decreasing of the adaptive coefficient of the last training to obtain the current adaptive coefficient can include: multiplying the adaptive coefficient of the last training by an inverse of the preset multiplier.

[0049] In some embodiments, the current regularization loss can be determined by: determining the current regularization loss based on the current training output result according to a second loss formula, the second loss formula including: wherein L is a number of the transformer layers in the to-be-trained model, T is a sequence length of the tokens corresponding to the training sample, and d is a number of all neurons in each mixed expert model. is a gating value corresponding to the nth neuron of the transformer layer of the lth layer and the tth token in the current training output result. is a gating value corresponding to the nth neuron of the transformer layer of the lth layer and the tth token in the current training output result.

[0050] In some embodiments, the current regularization loss can be determined by: determining the current regularization loss based on the current training output result according to a third loss formula, the third loss formula including: , L is the number of transformer layers in the model to be trained, T is the sequence length of the token corresponding to the training sample, d is the number of all neurons in each hybrid expert model, is the gating value corresponding to the n-th neuron of the t-th token in the transformer layer of the l-th layer in the current training output result, is the gating value corresponding to the n-th neuron of the t-th token in the transformer layer of the l-th layer in the current training output result, is the average activation ratio weight of the n-th neuron in the transformer layer of the l-th layer determined based on the current training output result, is the average activation ratio weight of the n-th neuron in the transformer layer of the l-th layer determined based on the current training output result,

[0051] In some embodiments, determining the average activation ratio weight can include determining the average activation ratio weight based on the current training output result according to an average activation ratio weight formula, the average activation ratio weight formula including: , k is the number of neurons expected to be activated in the hybrid expert model.

[0052] In some embodiments, the predictor can be a low-rank predictor network.

[0053] In some embodiments, the model projection layer can include an upward projection layer, a gating projection layer, and a downward projection layer, and the projection calculation unit 303 can be further configured to: obtain first projection results and second projection results obtained after the feature vector passes through the upward projection layer and the gating projection layer, respectively; multiply the gating values with the first projection results and the second projection results, respectively, to obtain first multiplication results and second multiplication results; multiply the first multiplication results and the second multiplication results to obtain intermediate features; based on the gating values, control the downward projection layer to selectively calculate the intermediate features to generate a projection calculation result.

[0054] In some embodiments, the projection calculation unit 303 can be further configured to: consider neurons in the model projection layer whose gating values are greater than a target threshold value as activated, and control the neurons considered to be activated in the model projection layer to perform sparse projection calculation on the feature vector. Here, the target threshold value is greater than 0.

[0055] Figure 4 A schematic diagram of a computer system that can implement the method and device for a hybrid expert model according to embodiments of the present application is shown. It should be understood that, Figure 4 The computer system 400 shown is only an example, and should not bring any limitation to the functions and use range of the method and device for a hybrid expert model according to embodiments of the present application.

[0056] As Figure 4As shown, the computer system 400 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 402 or loaded into a random access memory (RAM) 403 from a storage device 408. Various programs and data required for the operation of the computer system 400 are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0057] Generally, the following devices can be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a camera, an accelerometer, a gyroscope, a sensor, and the like; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, a motor, an electronic speed controller, and the like; a storage device 408 including, for example, a flash card, and the like; and a communication device 409. The communication device 409 can allow the computer system 400 to communicate with other devices wirelessly or through wires to exchange data. Although Figure 4 The computer system 400 is shown with various devices, but it should be understood that not all of the shown devices are required to implement or possess. More or fewer devices can alternatively be implemented or possessed. Figure 4 Each block shown in the flowcharts can represent one device or, as desired, multiple devices.

[0058] In particular, the processes described above with reference to the flowcharts can be implemented as a computer program according to some embodiments of the present application. For example, a computer readable medium is provided, on which a computer program is stored, the computer program containing instructions for performing the functions of the above-described functional units defined in the device for mixing expert models according to embodiments of the present application. Figure 2 The program code for the method of mixing expert models is shown. In such embodiments, the computer program can be downloaded and installed from a network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-described functional units defined in the device for mixing expert models according to embodiments of the present application are implemented.

[0059] Note that a computer readable medium according to an embodiment of the present application can be a computer readable signal medium or a computer readable storage medium or any combination thereof. The computer readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer readable storage medium of an embodiment of the present application can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. Further, a computer readable signal medium can include a computer readable program code carried by a computer readable signal, where the computer readable program code is embodied as a data signal propagated in a baseband or propagated as a carrier wave in a propagated signal. The computer readable signal medium can take a variety of forms, including but not limited to, a propagated data signal, an electrical signal, an optical signal, or any suitable combination of the foregoing. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate or propagate program code, such as a computer readable program code, for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0060] Computer program code for carrying out operations of embodiments of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0061] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0062] This invention can be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithm described in a particular embodiment can be modified without departing from the basic spirit of the invention. Therefore, the present embodiments are to be regarded as exemplary rather than limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and scope of the claims and their equivalents are thus included within the scope of the invention.

Claims

1. A method for mixing an expert model, the mixing expert model comprising a predictor, a gating module and a model projection layer, the method comprising: inputting a feature vector representing a token into the predictor to cause the predictor to generate importance scores corresponding to neurons in the model projection layer based on the feature vector; passing the importance scores through the gating module based on a rectified linear unit to generate gating values indicating whether neurons in the model projection layer are activated; and controlling the model projection layer to perform sparse projection computation on the feature vector based on the gating values to generate a projection computation result corresponding to the feature vector. The mixing expert model is obtained by joint training, each training step in the joint training comprising:

2. The method of claim 1, wherein, obtaining a current training output result of a training sample forward propagated in a to-be-trained model, the to-be-trained model comprising at least one transformer layer, wherein at least one transformer layer implements feedforward computation thereof by one of the mixing expert models; updating the to-be-trained model in a reverse direction using a current total loss. According to a first loss formula, a current total loss is determined based on a current training output result, the first loss formula being: , is a current total loss, is a current language model loss corresponding to the current training output result, is a current regularization loss about an activation sparsity corresponding to the current training output result, the activation sparsity being a ratio of a number of neurons that are not activated in the model to be trained to a total number of all neurons, λ is a current adaptive coefficient about the activation sparsity corresponding to the current training output result, and λ is a positive value; and A current adaptive coefficient is determined by:

3. The method of claim 2, wherein, determining a current average sparsity based on the current training output result; in a case where the current average sparsity is lower than a target sparsity, increasing an adaptive coefficient of a last training to obtain the current adaptive coefficient; and in a case where the current average sparsity is higher than the target sparsity, decreasing the adaptive coefficient of the last training to obtain the current adaptive coefficient. Determining the current average sparsity comprises: determining the current average sparsity based on the current training output result according to an average sparsity formula, the average sparsity formula comprising:

4. The method of claim 3, wherein, Increasing the adaptive coefficient of the last training to obtain the current adaptive coefficient comprises multiplying the adaptive coefficient of the last training by a preset multiplier, the preset multiplier being greater than 1; Decreasing the adaptive coefficient of the last training to obtain the current adaptive coefficient comprises multiplying the adaptive coefficient of the last training by an inverse of the preset multiplier. ， Wherein S is the current average sparsity, L is the number of transformer layers in the to-be-trained model, T is the sequence length of the token corresponding to the training sample, d is the number of all neurons in each mixed expert model, is the current training output result is the gate value corresponding to the nth neuron of the transformer layer of the tth token.

5. The method of claim 3, wherein, The current regularization loss is determined by: The current regularization loss is determined by:

6. The method of claim 2, wherein, The average activation ratio weight is determined by: According to a second loss formula, a current regularization loss is determined based on a current training output result, the second loss formula comprising: , L is a number of transformer layers in the model to be trained, T is a sequence length of a word piece corresponding to the training sample, d is a number of all neurons in each of the mixed expert models, is a gating value corresponding to an n-th neuron of a transformer layer of an l-th mixed expert model, and is a gating value corresponding to an n-th neuron of a transformer layer of an l-th mixed expert model, and 7. The method of claim 2, wherein, The predictor is a low-rank predictor network. According to a third loss formula, a current regularization loss is determined based on a current training output result, the third loss formula comprising: , L is a number of transformer layers in the to-be-trained model, T is a sequence length of a word piece corresponding to the training sample, d is a number of all neurons in each of the mixed expert models, is a current training output result, and is a gating value corresponding to an nth neuron of a transformer layer of an lth expert model in the mixed expert model, is an average activation ratio weight of the nth neuron of the transformer layer of the lth expert model determined based on the current training output result, and is a gating value corresponding to the nth neuron of the transformer layer of the lth expert model in the mixed expert model.

8. The method of claim 7, wherein, The model projection layer comprises an upward projection layer, a gating projection layer, and a downward projection layer, and controlling the model projection layer to perform sparse projection computation on the feature vector to generate a projection computation result corresponding to the feature vector comprises: The average activation ratio weight is determined based on a current training output result according to an average activation ratio weight formula, the average activation ratio weight formula comprising: k is a number of neurons in the hybrid expert model expected to be activated.

9. The method of claim 1, wherein, obtaining first and second projection results of the feature vector after passing through the upward projection layer and the gating projection layer, respectively; 10. The method of claim 1, wherein, multiplying the gating values with the first and second projection results, respectively, to obtain first and second multiplication results; multiplying the first and second multiplication results to obtain an intermediate feature; and controlling the downward projection layer to perform selective computation on the intermediate feature based on the gating values to generate the projection computation result. Controlling the model projection layer to perform sparse projection computation on the feature vector comprises: ​ 11. The method of claim 1, wherein, ​ The model projection layer is considered to be activated when the gating value of the neuron in the model projection layer is greater than a target threshold value, and the neuron in the model projection layer considered to be activated is controlled to perform sparse projection calculation on the feature vector, wherein the target threshold value is greater than 0.

12. A system for mixing an expert model, the mixing expert model comprising a predictor, a gating module and a model projection layer, the system characterized by comprising: an importance score unit configured to input a feature vector representing a token into the predictor to enable the predictor to generate an importance score corresponding to each neuron in the model projection layer based on the feature vector; a gating value unit configured to pass the importance score through the gating module based on a rectified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated; and a projection calculation unit configured to control the model projection layer to perform sparse projection calculation on the feature vector based on the gating value to generate a projection calculation result corresponding to the feature vector.

13. An apparatus for mixing expert models, the apparatus comprising: comprise: a processor; and a memory having computer-executable instructions stored thereon, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 11. The computer-executable instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 11.

14. A computer-readable storage medium having stored thereon computer- executable instructions, wherein, The computer-executable instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 11.

15. A computer program product comprising computer executable instructions, characterised in that, The computer-executable instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 11.

Images