Methods, systems, apparatus, media, and products for mixing expert models

Abstract

This invention discloses a method, system, apparatus, medium, and product for a hybrid expert model. The hybrid expert model includes a predictor, a gating module, and a model projection layer. The method includes: inputting feature vectors representing words into the predictor, causing the predictor to generate importance scores for each neuron in the corresponding model projection layer based on the feature vectors; passing the importance scores through a gating module based on modified linear units to generate gating values ​​indicating whether each neuron in the model projection layer is activated; and controlling the model projection layer to perform sparse projection calculations on the feature vectors based on the gating values ​​to generate projection calculation results for the corresponding feature vectors.

Description

Technical Field

[0001] This invention relates to the field of large language models, and more particularly to a method, system, apparatus, medium, and product for hybrid expert models. Background Technology

[0002] Mixture of Experts (MoE) models combined with activation sparsity techniques have become an important direction for accelerating large language models. Existing activation sparsity techniques employ a fixed selection strategy, using the same sparsity rate for all input tokens regardless of their complexity. This leads to simple tokens wasting unnecessary computational resources in the model, while complex tokens may suffer from a loss of accuracy due to insufficient activation neurons. Summary of the Invention

[0003] According to an embodiment of the present invention, a method for a hybrid expert model, the hybrid expert model including a predictor, a gating module, and a model projection layer, the method includes: inputting feature vectors representing words into the predictor, so that the predictor generates importance scores for each neuron in the corresponding model projection layer based on the feature vectors; passing the importance scores through a gating module based on a modified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated; and controlling the model projection layer to perform sparse projection calculation on the feature vectors based on the gating value, thereby generating a projection calculation result for the corresponding feature vectors.

[0004] According to an embodiment of the present invention, a system for a hybrid expert model includes a predictor, a gating module, and a model projection layer. The system includes: an importance score unit configured to input feature vectors representing words into the predictor, so that the predictor generates importance scores for each neuron in the corresponding model projection layer based on the feature vectors; a gating value unit configured to pass the importance scores through a gating module based on a modified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated; and a projection calculation unit configured to control the model projection layer to perform sparse projection calculation on the feature vectors based on the gating value, thereby generating a projection calculation result for the corresponding feature vectors.

[0005] An apparatus for a hybrid expert model according to an embodiment of the present invention includes: a processor; and a memory storing computer-executable instructions thereon, wherein the computer-executable instructions, when executed by the processor, cause the processor to perform the method described above.

[0006] According to an embodiment of the present invention, a computer-readable storage medium stores computer-executable instructions thereon, wherein, when executed by a processor, the computer-executable instructions cause the processor to perform the above-described method.

[0007] A computer program product according to an embodiment of the present invention includes computer-executable instructions, wherein, when executed by a processor, these computer-executable instructions cause the processor to perform the method described above. Attached Figure Description

[0008] The invention can be better understood from the following description of specific embodiments of the invention in conjunction with the accompanying drawings, wherein:

[0009] Figure 1 A schematic diagram of the structure of a hybrid expert model according to an embodiment of the present invention is shown.

[0010] Figure 2 A flowchart of a method for hybrid expert models according to an embodiment of the present invention is shown.

[0011] Figure 3 A schematic block diagram of a system for hybrid expert models according to an embodiment of the present invention is shown.

[0012] Figure 4 A schematic diagram of a computer system that can implement the method and apparatus for hybrid expert models according to embodiments of the present invention is shown. Detailed Implementation

[0013] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. Numerous specific details are set forth in the following detailed description to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention. The invention is by no means limited to any specific configurations and algorithms presented below, but covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of the invention. Well-known structures and techniques are not shown in the drawings and the following description in order to avoid unnecessarily obscuring the invention.

[0014] Typically, large language models consist of dozens or even hundreds of transformer layers. In the MoE architecture's large language model, some of the standard feed-forward networks (FFN) sublayers of the transformer layers are replaced by a hybrid expert layer. Each hybrid expert layer consists of a router and multiple hybrid expert models.

[0015] A word is the smallest unit of text input to a model. A word segmenter can process a sentence to obtain multiple words. After processing by at least one of the embedding layer or transformer layer in a large language model, a high-dimensional, dense feature vector is obtained. This feature vector contains the semantic and syntactic information of the word, as well as its relationship with the context, and can represent the original word.

[0016] When the feature vector corresponding to a word is sent to a hybrid expert model, the feature vector is multiplied by the weight matrix of the model's projection layer, and the result is used as the output of the hybrid expert model. This matrix multiplication operation accounts for the main computational cost in model inference. To improve computational efficiency while maintaining model performance, a combined scheme of hybrid expert models and activation sparsity techniques is proposed. This approach sparsely controls the neurons involved in computation, meaning that only a portion of the neurons in the model are activated and participate in computation, thereby accelerating model inference.

[0017] Existing activation sparsity techniques, such as the TopK selection strategy, employ a fixed selection policy, using the same sparsity rate (e.g., retaining 25% of neurons) for all input terms regardless of their complexity. This results in simple terms wasting unnecessary computational resources in the model, while complex terms may suffer from a loss of accuracy due to insufficient activation of neurons.

[0018] According to embodiments of the present invention, a predictor and a gating module based on the Rectified Linear Unit (ReLU) are configured in the hybrid expert model. When the feature vector of an indicator word is input to the predictor, the predictor outputs an importance score. This importance score is then passed through the Rectified Linear Unit-based gating module to obtain a gating value, which is used for sparse projection calculation of the model's projection layer. The trained predictor dynamically outputs corresponding importance scores based on different feature vectors. The gating value obtained through the gating module can activate different numbers of neurons. Therefore, the activation sparsity rate of the model's projection layer in the hybrid expert model dynamically changes according to the input feature vector, overcoming the shortcomings of the original fixed sparsity rate. The Rectified Linear Unit-based gating module has the advantages of continuous operation, full differentiability, and unbiased gradient propagation, thus providing differentiable control for the application of the hybrid expert model.

[0019] Figure 1 A schematic diagram of the structure of a hybrid expert model 100 according to an embodiment of the present invention is shown. The hybrid expert model 100 includes a predictor, a gating module, and a model projection layer. The model projection layer may include an upward projection layer, a gated projection layer, and a downward projection layer.

[0020] Figure 2 A flowchart of a dynamic control method 200 for intra-model sparsity according to an embodiment of the present invention is shown. Figure 2 As shown, the dynamic control method 200 includes steps S201-S203. S201: The feature vector representing the word is input into the predictor, so that the predictor generates importance scores for each neuron in the corresponding model projection layer based on the feature vector. S202: The importance scores are passed through a gating module based on a modified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated. S203: Based on the gating value, the model projection layer is controlled to perform sparse projection calculation on the feature vector, generating the projection calculation result of the corresponding feature vector.

[0021] In the method according to embodiments of the present invention, the predictor can be a low-rank predictor network, which can be represented as: ;in , , Let r and d be the dimensions of the input feature vector, respectively, and r and d correspond to different dimensions. Thus, the predictor exhibits a low-rank dimension.

[0022] like Figure 1 As shown, assuming a feature vector x, the importance scores of each neuron in the corresponding projection layer of the model obtained by inputting the predictor can be expressed as: The importance score for each neuron represents the predictor's initial assessment of the neuron's contribution to the model's output under the current input. The importance scores are then passed through a gating module based on modified linear units to generate gate values, which can be expressed as... Here, the gating value Each value in the graph represents whether the corresponding neuron is activated: a value of 0 indicates that the corresponding neuron is not activated; a value greater than 0 indicates that the corresponding neuron is activated. Specifically, the overall activation status of all neurons can be described by the instantaneous activation sparsity of the current model projection layer, which is the ratio of the number of currently unactivated neurons to the total number of neurons. The trained predictor will dynamically output corresponding importance scores based on different feature vectors x. The gating value obtained through the gating module can activate different neurons. Therefore, the activation sparsity of the model projection layer changes dynamically according to the input feature vector, overcoming the shortcomings of the original fixed sparsity.

[0023] like Figure 1As shown, in some embodiments, controlling the projection layer of the control model to perform sparse projection calculation on the feature vector to generate the projection calculation result of the corresponding feature vector may include: obtaining the first projection result and the second projection result obtained after the feature vector passes through the upward projection layer and the gated projection layer respectively; multiplying the gate value with the first projection result and the second projection result element by element to obtain the first multiplication result and the second multiplication result; multiplying the first multiplication result and the second multiplication result element by element to obtain the intermediate feature; and controlling the downward projection layer to selectively calculate the intermediate feature based on the gate value to generate the projection calculation result. The second projection result can also be processed by an activation function (e.g., a Sigmoid linear unit, SiLU) before performing element-wise multiplication. When the feature vector passes through the upward projection layer or the gated projection layer, it actually performs matrix multiplication with the weight matrix of the upward projection layer or the gated projection layer, and the result is the first projection result or the second projection result. Here, controlling the downward projection layer to perform selective calculation means only calculating the rows where the corresponding value in the gate value is greater than 0. The calculation specifically involves matrix multiplication of the weight matrix of the downward projection layer and the intermediate feature. Assume the weight matrix of the downward projection layer is:

[0024] ,

[0025] intermediate features are If the gate value g = [0, 0.3, 0, 0.8] at this time, then the intermediate features... and The value is 0. In this case, selective computation only involves the activation portion with a gating value greater than 0. The calculation process is as follows:

[0026] ;

[0027] .

[0028] In the method according to an embodiment of the present invention, the hybrid expert model can be deployed directly through inference, that is, by directly using the dynamic gating value generated by the gating module based on the modified linear unit. By default, neurons with a gating value greater than 0 are considered to be activated, so that the expert model maintains dynamic sparsity characteristics and is suitable for scenarios that are not sensitive to latency.

[0029] In the method according to embodiments of the present invention, the hybrid expert model can also be deployed for inference with a fixed target threshold. Specifically, in some embodiments, controlling the model projection layer to perform sparse projection calculation on the feature vector may include: considering neurons in the model projection layer with a gating value greater than the target threshold as activated, and controlling the neurons considered as activated in the model projection layer to perform sparse projection calculation on the feature vector. Here, the target threshold is greater than 0. Compared to the default activation condition where the gating value of the neuron is greater than 0 when directly deploying the hybrid expert model for inference, the comparison between the target threshold and the gating value further increases the threshold for neuron activation. This deployment scheme is suitable for hardware deployments requiring a fixed amount of computation.

[0030] In the method according to an embodiment of the present invention, the hybrid expert model can be obtained through joint training. Each training step in the joint training may include: obtaining the current training output result of the training sample in the model to be trained through forward propagation, determining the current total loss based on the current training output result according to the first loss formula, and using the current total loss to perform reverse update of the model to be trained.

[0031] The model to be trained includes at least one converter layer, wherein the at least one converter layer performs its feedforward computation through a hybrid expert model.

[0032] The first loss formula includes: . This represents the current total loss. The current language model loss corresponds to the current training output. The current language model loss can be determined based on methods known in the field of large language models, which will not be described in detail in this paper. This represents the current regularization loss with respect to the activation sparsity rate corresponding to the current training output. Activation sparsity is the ratio of the number of unactivated neurons in the model to the total number of neurons. Since directly training the gating module often results in a low sparsity rate (the model tends to activate more neurons to increase capacity), this embodiment introduces a regularization loss to force a higher sparsity rate. λ is the current adaptive coefficient with respect to the activation sparsity rate corresponding to the current training output, and λ is a positive value.

[0033] In some embodiments, the current adaptive coefficient can be determined by: determining the current average sparsity based on the current training output; increasing the adaptive coefficient of the previous training to obtain the current adaptive coefficient if the current average sparsity is lower than the target sparsity; and decreasing the adaptive coefficient of the previous training to obtain the current adaptive coefficient if the current average sparsity is higher than the target sparsity.

[0034] In some embodiments, determining the current average sparsity rate may include: determining the current average sparsity rate based on the current training output according to the average sparsity rate formula, wherein the average sparsity rate formula includes:

[0035] ,

[0036] Where S is the current average sparsity rate, L is the number of transformer layers in the model to be trained, T is the sequence length of the corresponding word in the training sample, and d is the total number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, the gating value corresponding to the t-th word in the n-th neuron, For conditions An indicator function that takes 1 if the value is true and 0 otherwise.

[0037] In some embodiments, increasing the adaptive coefficient from the previous training iteration to obtain the current adaptive coefficient may include multiplying the adaptive coefficient from the previous training iteration by a preset multiplier, where the preset multiplier is greater than 1. Correspondingly, decreasing the adaptive coefficient from the previous training iteration to obtain the current adaptive coefficient may include multiplying the adaptive coefficient from the previous training iteration by the reciprocal of the preset multiplier.

[0038] Therefore, assuming this training is the i-th iteration, the calculation of the current adaptive coefficients can be expressed as: .in, These are the current adaptive coefficients. These are the adaptive coefficients from the previous training iteration, and their initial values. It can be set to a very small positive number, for example 1-k / d is the target sparsity, which can be set to 75% for example. The preset multiplier is greater than 1, for example, it can be 1.2. This is a multiplier determined based on the relationship between the target sparsity and the current average sparsity, multiplied by the adaptive coefficients from the previous training iteration. Specifically, if the current average sparsity is lower than the target sparsity, the multiplier is... To increase the adaptive coefficient; if the current average sparsity is higher than the target sparsity, the multiplier is... The reciprocal of is used to reduce the adaptive coefficient.

[0039] In the method according to an embodiment of the present invention, the adaptive coefficients of the previous training are adjusted based on the relationship between the target sparsity rate and the current average sparsity rate to obtain the current adaptive coefficients. Other adjustment methods can also be used, such as using different calculation methods or different values, as long as the current adaptive coefficients meet the adjustment target.

[0040] In some embodiments, the current regularization loss can be determined by: determining the current regularization loss based on the current training output according to a second loss formula, wherein the second loss formula includes: L represents the number of converter layers in the model to be trained, T represents the sequence length of the corresponding word units in the training samples, and d represents the total number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, and the gating value corresponding to the t-th word in the n-th neuron.

[0041] Here, the second loss formula is actually the L1 regularization loss, which can be expressed as:

[0042] .

[0043] Due to the non-negativity of the output of the modified linear unit, the L1 norm is equal to the direct summation, thus yielding the second loss formula as shown above.

[0044] In some embodiments, to achieve load balancing among neurons, the calculation of regularization loss can incorporate an average activation ratio weight. In this case, the current regularization loss can be determined as follows: Based on the current training output, the current regularization loss is determined according to the third loss formula, which includes: , This represents the current regularization loss considering load balancing, where L is the number of transformer layers in the model to be trained, T is the sequence length of the corresponding word units in the training samples, and d is the total number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, the gating value corresponding to the t-th word in the n-th neuron, For the nth neuron determined based on the current training output, in the... Average activation ratio weight of the layer converter layer.

[0045] In some embodiments, the average activation ratio weight can be determined by: determining the average activation ratio weight based on the current training output according to the average activation ratio weight formula, wherein the average activation ratio weight formula includes: , where k is the number of neurons expected to be activated in the hybrid expert model. Here, this weight serves as the coefficient for the corresponding gating value, modifying the gradient of non-zero outputs to... This mechanism penalizes neurons that receive a relatively large number of terms, achieving load balancing by driving their gating values ​​toward zero more quickly. Unlike TopK routing, the gating value output by the modified linear unit-based gating module in this embodiment can be arbitrarily small, making... The value approaches 0. Therefore, in this embodiment, the adaptive coefficient λ is not a fixed value to avoid the route collapsing to 0. Thanks to the adaptive update of the adaptive coefficient λ, sparsity control and load balancing can be balanced in a single formula.

[0046] Based on the description of each training step in the joint training process described above, the entire joint training process can be divided into three stages according to performance: warm-up, sparsity phase, and stabilization phase. In the warm-up phase, the adaptive coefficient λ maintains a small initial value, and the model initializes training in a near-dense feedforward network manner, allowing the predictor to initially learn the feature representation of the input. In the sparsity phase, λ adaptively increases based on the deviation between the current average sparsity rate and the target sparsity rate, driving the model's average sparsity rate to gradually approach the preset target sparsity rate. The predictor then learns the optimal sparsity pattern corresponding to different inputs. Finally, in the stabilization phase, λ fluctuates and converges to a dynamic equilibrium point that stably maintains the average sparsity rate near the target sparsity rate. The model continues to optimize under this sparsity constraint, achieving efficient and stable convergence. These three stages ensure the smoothness and effectiveness of the co-evolution of the predictor and the model projection layer under sparsity constraints.

[0047] The method according to embodiments of the present invention implements multi-level dynamic computation allocation. First, there is word-level dynamics, where different numbers of neurons are activated based on the complexity of different words. For simple words, fewer neurons are activated to save computation; for complex words, more neurons are activated to ensure accuracy. Next, there is model-level dynamics, where different layers can learn different sparsity patterns. Shallow layers may perform more intensive computation to extract basic features, while deeper layers may be sparser and perform high-level semantic processing. Finally, there is expert-level dynamics, where different experts can have different sparsity characteristics; for example, general experts correspond to lower sparsity rates and handle common patterns, while specialized experts correspond to higher sparsity rates and handle specific domains.

[0048] The method according to embodiments of the present invention achieves significant improvements in multiple dimensions compared to existing solutions.

[0049] Regarding training efficiency, the non-differentiability of the original TopK operation leads to training difficulties. While the Balanced TopK module after the predictor output alleviates the neuron degeneration problem, TopK itself remains a discrete, non-differentiable operation. During backpropagation, gradients can only be approximated through a Straight-Through Estimator (STE), resulting in biased gradient estimation. This makes it difficult for the predictor to learn the optimal sparse pattern, leading to slow training convergence and ultimately limited performance. According to an embodiment of the present invention, the fully differentiable design of the hybrid expert model ensures accurate gradient propagation, enabling the predictor to learn the optimal sparse pattern more quickly, improving convergence speed by approximately 20-30%.

[0050] Regarding model accuracy, existing technologies lack dynamic computational allocation at the hierarchical and token levels. In existing schemes, the sparsity rate of each layer and each token is preset and fixed, making it impossible to dynamically adjust the allocation of computational resources according to the complexity of the actual input. According to embodiments of the present invention, the dynamic sparsity mechanism allows complex tokens to call more computational resources, effectively avoiding the information loss caused by fixed sparsity, and reducing model perplexity by 5-10% at the same sparsity rate.

[0051] Regarding computational resource utilization, the method according to embodiments of the present invention can automatically allocate computational load based on input complexity, enabling highly sparse computation for simple terms, further reducing the overall computational load by 10-20%, while maintaining or even improving model accuracy. In terms of deployment and application, the method according to embodiments of the present invention provides two inference modes: dynamic sparsity and fixed-threshold sparsity. The optimal deployment strategy can be flexibly selected based on actual hardware constraints and latency requirements, significantly enhancing the system's practicality and adaptability.

[0052] Figure 3 A schematic block diagram of a system 300 for a hybrid expert model according to an embodiment of the present invention is shown. Figure 3 As shown, the system 300 for a hybrid expert model includes an importance score unit 301, a gating value unit 302, and a projection calculation unit 303. The importance score unit 301 is configured to input feature vectors representing words into a predictor, enabling the predictor to generate importance scores for each neuron in the model's projection layer based on the feature vectors. The gating value unit 302 is configured to pass the importance scores through a gating module based on a modified linear unit to generate gating values ​​indicating whether each neuron in the model's projection layer is activated. The projection calculation unit 303 is configured to control the model's projection layer to perform sparse projection calculations on the feature vectors based on the gating values, generating projection calculation results for the corresponding feature vectors.

[0053] In some embodiments, the hybrid expert model is obtained through joint training. Each training step in the joint training may include: obtaining the current training output result of the training samples propagating forward in the model to be trained, wherein the model to be trained includes at least one transformer layer, and the at least one transformer layer implements its feedforward calculation through a hybrid expert model; determining the current total loss based on the current training output result according to a first loss formula, wherein the first loss formula is as follows: , This represents the current total loss. The loss of the current language model corresponds to the current training output. The current regularized loss with respect to activation sparsity is the ratio of the number of unactivated neurons to the total number of neurons in the model to be trained. λ is the current adaptive coefficient with respect to activation sparsity corresponding to the current training output, and λ is a positive value. The current total loss is used to back-update the model to be trained.

[0054] In some embodiments, the current adaptive coefficient can be determined by the following steps: determining the current average sparsity based on the current training output; increasing the adaptive coefficient of the previous training to obtain the current adaptive coefficient if the current average sparsity is lower than the target sparsity; and decreasing the adaptive coefficient of the previous training to obtain the current adaptive coefficient if the current average sparsity is higher than the target sparsity.

[0055] In some embodiments, determining the current average sparsity rate may include: determining the current average sparsity rate based on the current training output according to the average sparsity rate formula, wherein the average sparsity rate formula includes: Where S is the current average sparsity rate, L is the number of transformer layers in the model to be trained, T is the sequence length of the corresponding word in the training sample, and d is the number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, and the gating value corresponding to the t-th word in the n-th neuron.

[0056] In some embodiments, increasing the adaptive coefficient of the previous training to obtain the current adaptive coefficient may include: multiplying the adaptive coefficient of the previous training by a preset multiplier, wherein the preset multiplier is greater than 1; decreasing the adaptive coefficient of the previous training to obtain the current adaptive coefficient may include: multiplying the adaptive coefficient of the previous training by the reciprocal of the preset multiplier.

[0057] In some embodiments, the current regularization loss can be determined by the following steps: determining the current regularization loss based on the current training output according to a second loss formula, wherein the second loss formula includes: L represents the number of converter layers in the model to be trained, T represents the sequence length of the corresponding word units in the training samples, and d represents the total number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, and the gating value corresponding to the t-th word in the n-th neuron.

[0058] In some embodiments, the current regularization loss can be determined by the following steps: determining the current regularization loss based on the current training output according to the third loss formula, which includes: L represents the number of converter layers in the model to be trained, T represents the sequence length of the corresponding word units in the training samples, and d represents the total number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, the gating value corresponding to the t-th word in the n-th neuron, For the nth neuron determined based on the current training output, in the... Average activation ratio weight of the layer converter layer.

[0059] In some embodiments, determining the average activation ratio weight may include: determining the average activation ratio weight based on the current training output according to the average activation ratio weight formula, wherein the average activation ratio weight formula includes: , where k is the number of neurons expected to be activated in the hybrid expert model.

[0060] In some embodiments, the predictor can be a low-rank predictor network.

[0061] In some embodiments, the model projection layer may include an upward projection layer, a gated projection layer, and a downward projection layer. The projection calculation unit 303 may be further configured to: obtain a first projection result and a second projection result obtained by passing the feature vector through the upward projection layer and the gated projection layer, respectively; multiply the gate value element-wise with the first projection result and the second projection result to obtain a first multiplication result and a second multiplication result; multiply the first multiplication result and the second multiplication result element-wise to obtain intermediate features; and based on the gate value, control the downward projection layer to selectively calculate the intermediate features to generate projection calculation results.

[0062] In some embodiments, the projection calculation unit 303 may be further configured to: consider neurons in the model projection layer with a gating value greater than a target threshold as activated, and control the neurons in the model projection layer considered as activated to perform sparse projection calculation on the feature vector. Here, the target threshold is greater than 0.

[0063] Figure 4 A schematic diagram of a computer system that can implement the method and apparatus for a hybrid expert model according to embodiments of the present invention is shown. It should be understood that... Figure 4 The computer system 400 shown is merely an example and should not impose any limitation on the functionality and scope of use of the methods and apparatus for hybrid expert models according to embodiments of the present invention.

[0064] like Figure 4As shown, the computer system 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the computer system 400. The processing device 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0065] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, cameras, accelerometers, gyroscopes, sensors, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, motors, electronic speed controllers, etc.; storage devices 408 including, for example, flash cards; and communication devices 409. Communication device 409 allows computer system 400 to communicate wirelessly or wiredly with other devices to exchange data. Although... Figure 4 A computer system 400 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead. Figure 4 Each box shown can represent a device or multiple devices as needed.

[0066] In particular, according to some embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer programs. For example, a computer-readable medium is provided having a computer program stored thereon, the computer program comprising methods for executing... Figure 2 The program code shown is for a method for a hybrid expert model. In such an embodiment, the computer program can be downloaded and installed from a network via communication device 409, or installed from storage device 408, or installed from ROM 402. When the computer program is executed by processing device 401, the functional units described above in the apparatus for a hybrid expert model according to an embodiment of the present invention are implemented.

[0067] It should be noted that the computer-readable medium according to embodiments of the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. A computer-readable storage medium according to embodiments of the present invention may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Additionally, a computer-readable signal medium according to embodiments of the present invention may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (Radio Frequency), etc., or any suitable combination thereof.

[0068] Computer program code for performing operations according to embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0070] This invention can be implemented in other specific forms without departing from its spirit and essential characteristics. For example, the algorithm described in a particular embodiment can be modified without departing from the basic spirit of the invention. Therefore, the present embodiments are to be regarded as exemplary rather than limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes falling within the meaning and scope of the claims and their equivalents are thus included within the scope of the invention.

Claims

1. A method for a hybrid expert model, the hybrid expert model comprising a predictor, a gating module, and a model projection layer, the method being characterized by comprising: The feature vector representing the word is input into the predictor so that the predictor generates importance scores for each neuron in the projection layer of the model based on the feature vector. The feature vector includes the semantic and syntactic information of the word and its relationship with the context. It is obtained by processing the word through at least one of the embedding layer and the transformation layer in the large language model. The word is the smallest text unit input to the model and is obtained by the word segmenter processing the sentence. The importance score is passed through the gating module based on the modified linear unit to generate a gating value indicating whether each neuron in the model's projection layer is activated; and Based on the gate value, the model projection layer is controlled to perform sparse projection calculation on the feature vector, generating the projection calculation result corresponding to the feature vector. The predictor is a low-rank predictor network, and the model projection layer includes an upward projection layer, a gated projection layer, and a downward projection layer. The model projection layer is controlled to perform sparse projection calculations on the feature vector, generating projection calculation results corresponding to the feature vector, including: Obtain the first projection result and the second projection result obtained after the feature vector passes through the upward projection layer and the gated projection layer, respectively; The second projection result is processed based on the sigmoid linear unit; The gate value is multiplied element-wise with the first projection result and the processed second projection result to obtain the first multiplication result and the second multiplication result; The intermediate feature is obtained by multiplying the first multiplication result and the second multiplication result element by element; and Based on the gate value, the downward projection layer is controlled to selectively calculate the intermediate features to generate the projection calculation result. The hybrid expert model is obtained through joint training, and each training step in the joint training includes: Obtain the current training output result of the training samples forward propagation in the model to be trained, wherein the model to be trained includes at least one converter layer, and the at least one converter layer implements its feedforward computation through a hybrid expert model; According to the first loss formula, the current total loss is determined based on the current training output. The first loss formula is as follows: , This represents the current total loss. The loss of the current language model corresponds to the current training output. Let λ be the current regularization loss with respect to the activation sparsity rate corresponding to the current training output, where activation sparsity is the ratio of the number of unactivated neurons to the total number of neurons in the model to be trained, and λ is the current adaptive coefficient with respect to the activation sparsity rate corresponding to the current training output, and λ is a positive value; and The model to be trained is updated in reverse using the current total loss.

2. The method of claim 1, wherein, The current adaptive coefficients are determined through the following steps: Determine the current average sparsity based on the current training output; If the current average sparsity is lower than the target sparsity, increase the adaptive coefficient of the previous training to obtain the current adaptive coefficient. as well as If the current average sparsity is higher than the target sparsity, reduce the adaptive coefficients from the previous training to obtain the current adaptive coefficients.

3. The method of claim 2, wherein, Determining the current average sparsity includes: The current average sparsity is determined based on the current training output according to the average sparsity formula, which includes: ， Where S is the current average sparsity rate, L is the number of transformer layers in the model to be trained, T is the sequence length of the word corresponding to the training sample, and d is the number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, and the gating value corresponding to the t-th word in the n-th neuron.

4. The method of claim 2, wherein, Increasing the adaptive coefficient of the previous training to obtain the current adaptive coefficient includes: multiplying the adaptive coefficient of the previous training by a preset multiplier, wherein the preset multiplier is greater than 1; Reducing the adaptive coefficients from the previous training iteration to obtain the current adaptive coefficients involves multiplying the adaptive coefficients from the previous training iteration by the reciprocal of the preset multiplier.

5. The method of claim 1, wherein, The current regularization loss is determined through the following steps: According to the second loss formula, the current regularization loss is determined based on the current training output. The second loss formula includes: L is the number of converter layers in the model to be trained, T is the sequence length of the corresponding word units in the training samples, and d is the number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, and the gating value corresponding to the t-th word in the n-th neuron.

6. The method of claim 1, wherein, The current regularization loss is determined through the following steps: The current regularization loss is determined based on the current training output according to the third loss formula, which includes: L is the number of converter layers in the model to be trained, T is the sequence length of the corresponding word units in the training samples, and d is the number of neurons in each hybrid expert model. The first in the current training output The layer converter layer, the gating value corresponding to the t-th word in the n-th neuron, For the nth neuron determined based on the current training output, in the... Average activation ratio weight of the layer converter layer.

7. The method of claim 6, wherein, The average activation ratio weight is determined through the following steps: The average activation ratio weights are determined based on the current training output according to the average activation ratio weight formula, which includes: , where k is the number of neurons expected to be activated in the hybrid expert model.

8. The method according to claim 1, characterized in that, Controlling the model projection layer to perform sparse projection calculation on the feature vector includes: Neurons in the model projection layer whose gating value is greater than the target threshold are considered to be activated, and the activated neurons in the model projection layer are controlled to perform sparse projection calculation on the feature vector, wherein the target threshold is greater than 0.

9. A system for a hybrid expert model, the hybrid expert model comprising a predictor, a gating module, and a model projection layer, the system characterized in that it comprises: An importance score unit is configured to input a feature vector representing a word into the predictor, so that the predictor generates an importance score for each neuron in the projection layer of the model based on the feature vector. The feature vector includes the semantic and syntactic information of the word and its relationship with the context. It is obtained by processing the word through at least one of the embedding layer and the transformation layer in the large language model. The word is the smallest text unit input to the model and is obtained by the word segmenter processing the sentence. A gating unit is configured to pass the importance score through the gating module based on a modified linear unit to generate a gating value indicating whether each neuron in the model projection layer is activated; and The projection calculation unit is configured to control the model projection layer to perform sparse projection calculation on the feature vector based on the gate value, and generate the projection calculation result corresponding to the feature vector. The predictor is a low-rank predictor network, the model projection layer includes an upward projection layer, a gated projection layer, and a downward projection layer, and the projection calculation unit is further configured as follows: Obtain the first projection result and the second projection result obtained after the feature vector passes through the upward projection layer and the gated projection layer, respectively; The second projection result is processed based on the sigmoid linear unit; The gate value is multiplied element-wise with the first projection result and the processed second projection result to obtain the first multiplication result and the second multiplication result; The intermediate feature is obtained by multiplying the first multiplication result and the second multiplication result element by element; and Based on the gate value, the downward projection layer is controlled to selectively calculate the intermediate features to generate the projection calculation result. The hybrid expert model is obtained through joint training, and each training step in the joint training includes: Obtain the current training output result of the training samples forward propagation in the model to be trained, wherein the model to be trained includes at least one converter layer, and the at least one converter layer implements its feedforward computation through a hybrid expert model; According to the first loss formula, the current total loss is determined based on the current training output. The first loss formula is as follows: , This represents the current total loss. The loss of the current language model corresponds to the current training output. Let λ be the current regularization loss with respect to the activation sparsity rate corresponding to the current training output, where activation sparsity is the ratio of the number of unactivated neurons to the total number of neurons in the model to be trained, and λ is the current adaptive coefficient with respect to the activation sparsity rate corresponding to the current training output, and λ is a positive value; and The model to be trained is updated in reverse using the current total loss.

10. An apparatus for mixing expert models, the apparatus comprising: include: processor; as well as A memory having stored thereon computer-executable instructions, wherein, when executed by the processor, the computer-executable instructions cause the processor to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon computer- executable instructions, wherein, When executed by a processor, the computer-executable instructions cause the processor to perform the method of any one of claims 1 to 8.

12. A computer program product comprising computer executable instructions, characterised in that, When executed by a processor, the computer-executable instructions cause the processor to perform the method of any one of claims 1 to 8.

Images