Structured pruning method of hybrid expert model and electronic device

Abstract

This specification provides a structured pruning method and electronic device for hybrid expert models. The method includes: determining a target expert sub-model to be pruned from multiple expert sub-models of a hybrid expert model, wherein each expert sub-model's feedforward network includes a first linear layer, a second linear layer, and a third linear layer, and the structural dimensions of the feedforward network include intermediate dimensions and hidden dimensions; pruning any target expert sub-model, including: determining an intermediate channel retention strategy for the target expert sub-model in the intermediate dimension and a hidden channel retention strategy in the hidden dimension; pruning the intermediate channels of the first, second, and third linear layers of the sub-model according to the intermediate channel retention strategy; pruning the hidden channels of the first and second linear layers of the sub-model according to the hidden channel retention strategy, while keeping the output dimension of the third linear layer unchanged.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of artificial intelligence (AI) and deep learning model compression technology, and more particularly to a structured pruning method and electronic device for hybrid expert models. Background Technology

[0002] As the parameter size of large-scale deep learning models (such as large language models LLMs, multimodal large models LMMs, etc., hereinafter referred to as large models) continues to expand, the computing power and storage costs required for model deployment and inference are increasing exponentially. The Mixture of Experts (MoE) architecture, as a conditional computation paradigm, introduces a routing network to dynamically allocate input data to activated expert sub-models, which significantly reduces the computational cost of a single inference while maintaining the total number of model parameters, and is therefore widely used.

[0003] However, even deep learning models using the MoE architecture (i.e., hybrid expert models, or MoE models) still have a large number of parameters, making them difficult to deploy directly on resource-constrained edge devices or online services with low latency requirements. Therefore, compressing hybrid expert models has become an urgent technical problem to be solved.

[0004] Among existing model compression techniques, pruning is an effective method, which can be divided into unstructured pruning and structured pruning according to different compression methods. Among them, structured pruning removes channels, filters, or entire linear layers (projections) to ensure that the weight matrix after pruning still maintains a regular tensor shape, thus enabling seamless integration with intensive computing hardware.

[0005] Existing structured pruning schemes generally fall into two categories: one is to structurally prune only the "intermediate dimension" of the expert sub-model's feed-forward network (FFN) to achieve model compression by reducing the spread width of the feed-forward network. This type of "pruning only the intermediate dimension" scheme is more effective for early models with a large FFN spread ratio. However, as the FFN spread ratio of the MoE models adopted in the industry has gradually decreased, the compression efficiency of this type of scheme has become difficult to improve further. The other type is a hidden dimension compression scheme designed for the standard Transformer architecture. This type of scheme usually relies on globally shared masks, additional alignment projections, or recovery operators to maintain residual compatibility after reducing the hidden layer width. However, because the expert sub-models of MoE usually exhibit heterogeneous activation distributions and redundant patterns, this type of scheme is not suitable for MoE architectures; moreover, this type of scheme requires additional computation and memory overhead during inference, thereby reducing deployment efficiency. Summary of the Invention

[0006] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a structured pruning method using a hybrid expert model is proposed, comprising: In a hybrid expert model, a target expert submodel to be pruned is determined from among multiple expert submodels. The feedforward network of each expert submodel includes a first linear layer, a second linear layer, and a third linear layer. The structural dimensions of the feedforward network include intermediate dimensions and hidden dimensions. Pruning the target expert sub-model; wherein pruning any target expert sub-model includes: Determine the intermediate channel retention strategy of any target expert sub-model in the intermediate dimension and the hidden channel retention strategy in the hidden dimension; For the first linear layer, second linear layer, and third linear layer of any target expert sub-model, the intermediate dimension channels are pruned according to the intermediate channel retention strategy. For the first and second linear layers of any target expert sub-model, the hidden dimensions are pruned according to the hidden channel retention strategy, while keeping the output dimension of the third linear layer unchanged.

[0007] According to a second aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in the first aspect by executing the executable instructions.

[0008] According to a third aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0009] According to a fourth aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0010] As can be seen from the above embodiments, the solution in this specification formulates channel preservation strategies for the intermediate and hidden dimensions of the feedforward network of each target expert sub-model to be pruned. Next, not only are the intermediate dimensions of the first, second, and third linear layers pruned according to the intermediate channel preservation strategy, but the hidden dimensions of the first and second linear layers are also pruned according to the hidden channel preservation strategy, while keeping the output dimension of the lower projection layer unchanged. Therefore, unlike related technologies that "prune only the intermediate dimension," this solution breaks through the limitation of a single dimension and achieves joint structured pruning of the intermediate and hidden dimensions, that is, it simultaneously performs structured pruning of the linear layers inside the feedforward network of the target expert sub-model from both the intermediate and hidden dimensions.

[0011] Understandably, this solution introduces a "hidden dimension channel preservation strategy" to prune the hidden dimensions of the first and second linear layers. This expands the compression dimension from a single "intermediate dimension" to a two-dimensional collaborative compression of "intermediate dimension + hidden dimension," enabling in-depth mining and removal of redundant features at the input interface of the MoE model, thereby significantly improving the model's parameter compression efficiency. In fact, because the pruning of the hidden dimension reduces the input width of the expert sub-model, combined with the pruning of the intermediate dimension, a compression rate far exceeding that of traditional single-dimensional solutions can be achieved without compromising the model's core inference capabilities. Moreover, this solution first determines the target expert sub-model to be pruned, then determines the intermediate channel preservation strategy and hidden channel preservation strategy for each target expert sub-model, and performs targeted structured pruning accordingly. This "expert-specific" adaptive channel preservation mechanism allows the pruning operation to accurately match the unique redundancy distribution within different expert sub-models, avoiding the accidental deletion of key features that might result from a globally uniform pruning ratio, thus effectively ensuring the inference performance of the pruned model. In addition, by "keeping the output dimension of the third linear layer unchanged", it is ensured that the pruned expert sub-model can still be aligned with the global residual connection of the MoE model at the output, without introducing any additional alignment projection or recovery operator. Therefore, not only is the hidden dimension compressed, but the regularity of the overall topology of the model is also maintained, ensuring that the pruned model can be seamlessly connected to existing intensive computing hardware and avoiding the reduction in deployment efficiency due to compression. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating an architectural pruning method for a hybrid expert model, as provided in an exemplary embodiment.

[0013] Figure 2 This is a schematic diagram of a pruning process for a hybrid expert model provided in an exemplary embodiment.

[0014] Figure 3 This is a schematic diagram illustrating the implementation process of a budget-constrained coordinate descent method as provided in an exemplary embodiment.

[0015] Figure 4 This is a flowchart of an exemplary embodiment of a reasoning method based on a hybrid expert model.

[0016] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment.

[0017] Figure 6 This is a block diagram of an architectural pruning device for a hybrid expert model, provided in an exemplary embodiment.

[0018] Figure 7 This is a block diagram of an exemplary embodiment of a hybrid expert model-based inference device. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0020] As mentioned earlier, structured pruning schemes in related technologies generally fall into two categories: one is to perform structured pruning only on the "intermediate dimension" of the expert sub-model feedforward network to achieve model compression by reducing the spread width of the feedforward network. This type of "pruning only the intermediate dimension" scheme is more effective for early models with a large FFN spread ratio. However, as the FFN spread ratio of the MoE models adopted in the industry has gradually decreased, the compression efficiency of this type of scheme has become difficult to improve further. The other type is a hidden dimension compression scheme designed for the standard Transformer architecture. This type of scheme usually relies on a globally shared mask, additional alignment projection, or recovery operator to maintain residual compatibility after reducing the hidden layer width. However, because the expert sub-models in the MoE model typically employ advanced activation function architectures such as SwiGLU, there are strict dimensional coupling relationships between the multiple linear layers (such as gated linear layers, up-projection linear layers, and down-projection linear layers) contained within their feedforward networks. Furthermore, the expert sub-models often exhibit heterogeneous activation distributions and redundant patterns. Therefore, such solutions are not suitable for the MoE architecture. Moreover, these solutions require additional computational and memory overhead during inference, thereby reducing deployment efficiency.

[0021] Figure 1 This is a flowchart illustrating an exemplary embodiment of an architectural pruning method for a hybrid expert model, which can be applied to training devices for hybrid expert models. Figure 1 As shown, the method includes the following steps 102-104.

[0022] Step 102: Determine the target expert sub-model to be pruned among the multiple expert sub-models contained in the hybrid expert model. The feedforward network of each expert sub-model contains a first linear layer, a second linear layer, and a third linear layer. The structural dimensions of the feedforward network include intermediate dimensions and hidden dimensions.

[0023] The training device first needs to acquire the hybrid expert model (i.e., the MoE model) to be pruned. This model contains multiple expert sub-models, and the feedforward network of each expert sub-model contains multiple linear layers. These multiple linear layers include at least a first linear layer, a second linear layer, and a third linear layer (and may also include other linear layers besides these three, which are not the focus of this embodiment). The structural dimensions of the feedforward network include intermediate dimensions and hidden dimensions. The intermediate dimensions and hidden dimensions can be two mutually orthogonal dimensions. For example, in the feedforward network of any expert sub-model, the input ends of the first and second linear layers belong to the hidden dimension (the hidden dimension is its input dimension), and the output ends belong to the intermediate dimension (i.e., the intermediate dimension is its output dimension); while in the feedforward network, the input end of the third linear layer belongs to the intermediate dimension (the intermediate dimension is its input dimension), and the output end belongs to the hidden dimension (i.e., the hidden dimension is its output dimension).

[0024] The linear layers described in this specification are fundamental components used to perform linear transformations on the input data. Mathematically, they perform an affine transformation, where the output equals the product of the input vector and the weight matrix, plus a bias vector. In the hybrid expert model, the three linear layers of each expert sub-model play different roles: The first linear layer, upon receiving the feature vector of the input token, can either directly connect to the input of the third linear layer, or output a probability distribution or weight vector through a linear transformation (which can be followed by Softmax or Top-K operations), and then connect this output to the input of the third linear layer. This weight vector determines which experts the current token should be sent to for processing, and the contribution percentage of each expert. This linear layer does not directly participate in the nonlinear transformation of features, but is responsible for "decision-making."

[0025] The second linear layer is responsible for feature upscaling and expansion, mapping the low-dimensional input feature vector to a higher-dimensional hidden space. This linear layer projects the input into a richer feature space, allowing subsequent activation functions to capture more complex nonlinear relationships. In MoE, each expert has their own independent upprojection weights, meaning different experts learn different feature expansion methods.

[0026] The third linear layer is responsible for feature compression and fusion, that is, remapping the feature vectors in the high-dimensional hidden space after activation function processing back to the original low-dimensional space. This linear layer is used to integrate specific knowledge or features extracted by experts, making their dimensions match the residual connections so that subsequent layers can continue processing. Similarly, the downprojection weights of each expert are also independent.

[0027] The inputs of the first and second linear layers can be connected to the input of the feedforward network, or connected to the input of the feedforward network through a convergence operator and a routing network. The outputs of the first and second linear layers can be connected to the input of the third linear layer, wherein the two outputs can be directly connected to the input of the third linear layer; or, the output of the first linear layer can be connected to the input of the third linear layer through an activation function, and the output of the second linear layer can be directly connected to the input of the third linear layer. This specification does not limit the connection relationships between the various linear layers in the embodiments.

[0028] In one embodiment, the first linear layer, second linear layer, and third linear layer included in the feedforward network of each expert sub-model can be the aforementioned gated linear layer, upward-projected linear layer, and downward-projected linear layer, respectively. To fully disclose the specific implementation of the scheme in this specification, the following embodiments will use gated linear layers, upward-projected linear layers, and downward-projected linear layers as examples for explanation. It is understood that the input terminals of the gated linear layer and the upward-projected linear layer can be connected to the input terminal of the feedforward network, or connected to the input terminal of the feedforward network through a convergence operator and a routing network. The output terminals of the gated linear layer and the upward-projected linear layer can be connected to the input terminal of the downward-projected linear layer, wherein the two output terminals can be directly connected to the input terminal of the downward-projected linear layer; or, the output terminal of the gated linear layer can be connected to the input terminal of the downward-projected linear layer through an activation function, and the output terminal of the upward-projected linear layer can be directly connected to the input terminal of the downward-projected linear layer. This specification does not limit the connection relationships between the various linear layers in the embodiments.

[0029] In the embodiments of this specification, before pruning the hybrid expert model, it is necessary to first determine which sub-models among the multiple expert sub-models need to be pruned; that is, it is necessary to first determine the target expert sub-models to be pruned. The pruning objects in each step below are the target expert sub-models determined in that step.

[0030] In one embodiment, the target expert sub-model to be pruned can be determined based on the importance of each expert sub-model. The importance score of any expert sub-model is used to quantitatively evaluate its contribution or redundancy to the inference task required by the hybrid expert model, i.e., to characterize its importance within the hybrid expert model. For example, where the importance score is positively correlated with its importance, a lower importance score indicates that the expert is less important or more redundant, and therefore more suitable for pruning. In this case, the importance score of each expert sub-model in the hybrid expert model can be calculated first, and then at least one expert sub-model with an importance score below a preset threshold, or a preset number (at least 1) of expert sub-models with the lowest importance scores, can be identified as the target expert sub-models to be pruned. This embodiment, through a quantitative scoring mechanism, can objectively measure the actual contribution of each expert sub-model, avoiding the subjectivity and inaccuracy of human experience judgment; secondly, by using threshold filtering or sorting selection methods, the granularity and scope of pruning can be flexibly controlled, ensuring that the least important redundant parts are eliminated first. Moreover, this method can significantly reduce the number of model parameters and computational overhead while preserving the feature extraction capabilities of core experts to the greatest extent and maintaining the high accuracy of the model.

[0031] The importance score of each expert sub-model can be calculated in various ways. For example, the importance score can be calculated based on the routing weights of the expert sub-models. If the hybrid expert model includes a routing network (the network is used to assign the probability of each token to different expert sub-models), the number of samples or routing probabilities of the validation set data being assigned to each expert sub-model after passing through the routing network can be counted, and the output feature contribution of each expert sub-model in processing the validation set data can be calculated. Then, based on the weight parameters of each expert sub-model, the corresponding absolute mean or root mean square importance index can be calculated, and the importance score of each expert sub-model can be calculated based on the above index. For example, assuming there are 1000 token inputs, expert A has a total weight of 500 when selected, while expert B has a total weight of only 10 when selected. Then, the importance score of expert B is much lower than that of expert A, indicating that expert B is in an "underactivated" state and can be used as a pruning target, that is, expert B can be identified as the target expert sub-model. It is evident that the more frequently any expert sub-model is called, the more general features it processes, and the higher its importance in the hybrid expert model.

[0032] For example, importance scores can also be calculated based on the gradient / sensitivity scores of expert sub-models. For any expert sub-model, the gradient norm of the model's loss function with respect to the expert's parameters can be calculated, or the change in the loss of the hybrid expert model after removing the expert can be calculated. In other words, the actual degree of intervention by the expert on the current token can be evaluated by calculating the L2 norm (Least Squares Error, LSE) of the expert module's output, or by calculating the change in similarity of the input hidden state before and after passing through the expert module. For instance, if the parameters of expert C are set to zero, the loss of the hybrid expert model hardly changes (low sensitivity), indicating that expert C has a low importance score. In this case, expert C can be identified as the target expert sub-model.

[0033] For example, importance scores can be determined based on the statistical values ​​of the weight parameters and / or activation values ​​of the expert sub-model. The statistical values ​​of the weight parameters may include the L1 or L2 norm of the weight parameters, and the statistical values ​​of the activation values ​​may include the average absolute value of the activation values ​​of each channel generated by the expert sub-model when processing a preset dataset. This method indicates that if an expert's weight values ​​are generally very small (close to 0), it means its contribution to the output is negligible, and it belongs to a "redundant expert" that can be pruned. Alternatively, the average absolute value of the activation values ​​of each channel of any expert sub-model can be used as the importance score of that sub-model. Or, even if an expert has a large weight, if the activation value generated when input data flows through it is small, it indicates that this expert is not effectively utilized in processing the current inference task corresponding to that input data, and therefore also belongs to a "redundant expert" that can be pruned.

[0034] Of course, besides determining the target expert sub-model to be pruned through the aforementioned importance scoring method, other methods can also be used. For example, the target expert sub-model can be determined based on cluster analysis. For instance, clustering analysis can be performed on the output vectors or weight vectors of all experts. If multiple experts are functionally highly similar (clustered in the same class and very close in distance), they are considered redundant, and the experts in this class, excluding the "central expert," can be determined as the target expert sub-model. Another example is determining the target expert sub-model based on random search or evolutionary algorithms. For instance, a group of experts can be randomly selected for pruning attempts, and the performance of the pruned model (such as accuracy and latency) can be evaluated. If the performance degradation is within acceptable limits, these experts can be determined as target expert sub-models; otherwise, the process is rolled back and the next group is tried until all experts have been searched. Yet another example is determining the target expert sub-model based on predefined rules (i.e., fixed strategies), such as according to the parity of expert IDs, or by selecting one expert every N experts as the target sub-model, etc., which will not be elaborated further.

[0035] Using the above method, at least one target expert sub-model to be pruned can be determined from the multiple expert sub-models contained in the hybrid expert model. Let's assume the hybrid expert model contains E (E is a positive integer greater than 1) expert sub-models, from which E1 (E1 is a positive integer) target expert sub-models are determined—these target expert sub-models are the pruning targets in subsequent step 104. Correspondingly, there may be E2 (E2 can be zero or a positive integer; if E2 is zero, then E1 = E, in which case all expert sub-models need to be / will be pruned) expert sub-models that do not need pruning—the subsequent pruning steps in this specification do not involve these expert sub-models and will not be elaborated upon further.

[0036] Step 104: Prune the target expert sub-model.

[0037] After identifying at least one target expert sub-model using the aforementioned method, each target expert sub-model can be pruned individually. Once all target expert sub-models have been pruned, the pruning of the hybrid expert model is complete. It should be noted that this scheme uses the complete channels of each linear layer as the basic unit for pruning to achieve compression of the linear layers; therefore, the "pruning" in this scheme is "structured pruning." Unlike unstructured pruning that randomly prunes individual weight parameters, the core of the structured pruning used in this scheme is to maintain the regularity of the model's topology, thereby ensuring that the pruned model can directly adapt to the parallel computing characteristics of hardware accelerators (such as GPUs / TPUs), avoiding irregular memory access and scheduling overhead caused by sparse matrices.

[0038] Figure 2 This is a schematic diagram illustrating a pruning process for a hybrid expert model, provided in an exemplary embodiment. Figure 2As shown, the MoE module of this model contains multiple expert sub-models, which can be divided into E1 target expert sub-models and E2 unpruned expert sub-models (such as E2 shared expert sub-models). Each expert sub-model has the same internal structure. Taking any target expert sub-model (i.e., the i-th expert Expert_i) as an example, the expert's FFN contains multiple linear layers, such as a gated linear layer (Gate_proj) as the first linear layer, an up-projected linear layer (Up_proj) as the second linear layer, and a down-projected linear layer (Down_proj) as the third linear layer. The output of the convergence operator in this FFN is connected to the input of the gated linear layer and the up-projected linear layer. The outputs of the gated linear layer and the up-projected linear layer are multiplied element-wise and then connected to the input of the down-projected linear layer (where the output of the gated linear layer is activated by the activation function Act_Fn before being multiplied element-wise with the output of the up-projected linear layer). The data flow within FFN is as follows: Input data → Gated linear layer determines which expert sub-model the data enters → After the data enters certain expert sub-models → Up projection linear layer increases the dimensionality of the data → The increased dimensionality features pass through the activation function → Down projection linear layer reduces the dimensionality of the activation results → Output data.

[0039] As shown on the right side of the internal structure of the FFN in the figure, before pruning, the intermediate dimension of any linear layer in the gated linear layer and the upprojected linear layer is dffn, and the hidden dimension is dmodel. After pruning the intermediate dimension of any linear layer (shown by the horizontal scissors icon), the intermediate dimension of the linear layer changes (i.e. shrinks) to dffn′; and after pruning the hidden dimension of any linear layer (shown by the vertical scissors icon), the hidden dimension of the linear layer changes (i.e. shrinks) to dmodel′. It can be seen that pruning the intermediate channels of the gated linear layer and the upprojected linear layer is actually pruning their outputs, thereby reducing the number of output channels; while pruning their hidden dimensions is actually pruning their inputs, thereby reducing the number of input channels.

[0040] Similarly, before pruning, the intermediate dimension of the downprojected linear layer is dffn, and the hidden dimension is dmodel. After pruning the intermediate channels of the downprojected linear layer (shown by the horizontal scissors icon), the intermediate dimension of the linear layer changes (i.e. shrinks) to dffn′. It can be seen that pruning the intermediate channels of the downprojected linear layer is actually pruning the input end of the linear layer, thereby reducing the number of input channels. Keeping the output dimension of the downprojected linear layer unchanged, that is, keeping the number of output channels unchanged, ensures that the output dimension of the pruned downprojected linear layer remains dmodel.

[0041] It should be noted that before pruning begins, given that the input data received by each expert sub-model comes from the same previous layer (i.e., the result obtained from residual connections and normalization), and that the outputs of each expert sub-model need to be converged to form the main loop in parallel pruning, the hidden dimension dmodel of the gated linear layer, the upprojected linear layer, and the downprojected linear layer must be the same to ensure that the matrices of each expert can be added element-wise. Furthermore, to utilize the parallel computing capabilities of the GPU (packaging multiple experts into a large matrix multiplication operation), the intermediate dimension dffn of these three linear layers can also be the same (of course, dffn can be different to meet certain specific engineering requirements). Similarly, after pruning, the hidden dimension dmodel′ of the three linear layers must be the same, while the intermediate dimension dffn′ can be the same or different; this specification does not impose any limitations on this in the embodiments.

[0042] It is worth noting that before pruning, if the intermediate dimension dffn and hidden dimension dmodel of each linear layer are the same, that is, the size / specification of each linear layer is consistent, then each channel of the intermediate dimension of each linear layer (i.e., each intermediate channel) can use the same row index, and each channel of the hidden dimension of each linear layer (i.e., each hidden channel) can use the same column index. The following example illustrates this using the same row index and column index for each linear layer.

[0043] The specific process of pruning each target expert sub-model is not fundamentally different. The following explanation uses the pruning process of any target expert sub-model as an example. The pruning process of any target expert sub-model may include the following steps 1042-1046.

[0044] Step 1042: Determine the intermediate channel retention strategy of any target expert sub-model on the intermediate dimension and the hidden channel retention strategy on the hidden dimension.

[0045] The "channel retention strategy" described in the embodiments of this specification refers to a rule or mapping relationship used to characterize which dimensions (channels) in a linear layer are retained and which are removed. From a computer implementation perspective, the channel retention strategy can be represented as a Boolean mask or an index list / set. Specifically, the intermediate channel retention strategy is used to characterize which channels of the intermediate dimensions of the linear layer are retained and which are removed; while the hidden channel retention strategy is used to characterize which channels of the hidden dimensions of the linear layer are retained and which are removed.

[0046] In one embodiment, the Budget-Constrained Coordinate Descent (BCD) method can be used to determine the intermediate channel retention strategy in the intermediate dimension and the hidden channel retention strategy in the hidden dimension for any target expert sub-model. BCD is an iterative pruning allocation algorithm that can find the optimal pruning ratio between the intermediate layer and the hidden dimension of any expert model under a fixed global parameter budget. For example, the total parameter budget of any target expert sub-model can be obtained first, representing the total number of parameters in the intermediate and hidden dimensions after pruning. Then, with the goal of minimizing model reconstruction error, the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension are alternately optimized under the constraint of the total parameter budget until convergence is met. Finally, based on the converged number of intermediate and hidden channels retained, corresponding intermediate channel retention strategies and hidden channel retention strategies are generated, respectively.

[0047] The total parameter budget can be considered as a resource constraint boundary condition when performing structured pruning on any of the target expert sub-models. Specifically, it can be a pre-set hard constraint index for the sub-model, specifying the upper limit of the expected / allowed maximum number of parameters for the pruned target expert sub-model. This budget can be a scalar value (e.g., N floating-point numbers), or it can be a proportion relative to the total number of parameters of the hybrid expert model before pruning or relative to the total number of parameters of the sub-models of any of the target expert sub-models before pruning (e.g., retaining 10% or 50% of the total number of parameters of the model before pruning, etc.). The embodiments in this specification do not impose such limitations. In addition, the model reconstruction error index is used to measure the difference between the "output of the model after pruning" and the "output of the model before pruning". For example, the same batch of validation set data can be input into the hybrid expert model before pruning and the hybrid expert model after pruning (hereinafter referred to as the pruned expert model), respectively, and the mean squared error (MSE) or L1 norm distance between the feature maps of the two at a specific linear layer output (or final output) can be calculated.

[0048] By utilizing the BCD algorithm, this embodiment can automatically find the balance point between intermediate and hidden dimensions under limited parameter resources, thus avoiding a significant drop in model accuracy caused by blind trial and error. First, by setting a total budget parameter, the compression rate of any target expert sub-model can be precisely controlled, ensuring that the pruned model strictly meets specific hardware deployment requirements or storage limitations. Second, by employing an alternating optimization approach to solve for the number of dimensions to retain, the combination of dimensions that minimizes reconstruction error can be found in a complex search space, achieving a higher accuracy retention rate than fixed-ratio pruning. Furthermore, transforming dimension selection into a constrained optimization problem avoids the tedious process of manually adjusting the width of each layer based on human experience, improving the automation level of model compression.

[0049] In one embodiment, when alternately optimizing the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension, the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension can be initialized first. Then, under the constraint of satisfying the total budget of the parameters, the search is switched between the number of intermediate channels retained and the number of hidden channels retained in an alternating iterative manner until the convergence condition is met and the search stops. In each search, either the number of intermediate channels retained or the number of hidden channels retained is fixed, and the number of the other channel retained is updated with minimizing the model reconstruction error as the objective function.

[0050] The switching search mentioned here refers to an optimization strategy based on coordinate descent. Finding the optimal solution for two variables (the intermediate dimension dffn′ and the hidden dimension dmodel′) simultaneously in a multidimensional parameter space is extremely difficult and computationally intensive. Therefore, this scheme decomposes the joint optimization problem into two univariate optimization subproblems: in each iteration, one variable A is fixed while the other variable B is adjusted to find the current optimal solution; in the next iteration, the process is reversed, fixing variable A at the optimal value obtained in the previous iteration and adjusting variable B. This "switching" continues until both variables no longer change significantly (i.e., convergence), at which point the iteration stops.

[0051] For example, suppose the total budget of any target expert sub-model is limited to 100 units, and the initial state is set as follows: the number of intermediate channels retained is Nffn=50, and the number of hidden channels retained is Nmodel=2 (assuming the parameter calculation formula is simplified to P≈Nffn×Nmodel).

[0052] First iteration (fixed Nmodel, search Nffn): Keep Nmodel=2 unchanged. Based on the budget constraint Nffn×2≤100, the maximum value of Nffn is derived to be 50. At this time, test the model reconstruction error when Nffn takes values ​​in the range [1,50]. It is found that the error is the smallest when Nffn=40 (possibly due to insufficient sparsity caused by excessive width or other regularization terms), so update Nffn=40.

[0053] Second iteration (fixed Nffn, search Nmodel): Keep Nffn=40 from the previous round unchanged. Based on the budget constraint 40×Nmodel≤100, the maximum value of Nmodel is 2.5 (rounded to 2). At this point, test the value of Nmodel within the allowed range. It is found that Nmodel=2 is still optimal for the current width, or if the budget allows for a larger value, it might be adjusted. Assuming that keeping Nmodel at 2 results in the lowest error, then update Nmodel to 2.

[0054] Third iteration (switching back to Nffn): Keep Nmodel=2 unchanged and check Nffn again. It is found that Nffn remains stable at 40 and no longer jumps. At this point, the convergence condition is met, and the final policy is output: Nffn=40, Nmodel=2.

[0055] This switching search method utilizes a budget-constrained coordinate descent approach to allocate the corresponding number of channels to the hidden and intermediate dimensions of any target expert sub-model, while satisfying the constraint of the total parameters of that sub-model. In other words, it dynamically adjusts the proportion of channels (or parameter proportions) in the hidden and intermediate dimensions of the expert sub-model while keeping the parameter budget of that sub-model fixed.

[0056] By decomposing the complex bivariate joint optimization into alternating univariate iterations, the dimensionality of the search space is significantly reduced, making it possible to quickly find an approximate global optimum under limited computing power. Secondly, the "fix one, optimize the other" switching mechanism effectively decouples the nonlinear dependencies between the two dimensions, avoiding getting trapped in local suboptimal solutions and improving the accuracy of the pruning strategy. Moreover, this method naturally adapts to the hard constraint of the total parameter budget, ensuring that the candidate solutions generated in each iteration are legal and feasible, requiring no additional post-processing correction steps. Furthermore, this stepwise convergence method exhibits good stability and can adapt to the sensitivity differences at different levels, thereby maximizing the preservation of key feature information while maintaining model compression ratio. Therefore, this approach effectively solves the problems of strong coupling between intermediate and hidden dimensions in the process of hybrid expert model pruning, which makes it difficult to directly solve for the optimal dimension combination using analytical formulas, and also results in an excessively large joint search space and extremely high computational cost.

[0057] It is understandable that the above-mentioned switching search method actually adopts a "strict alternation" optimization logic. In addition, alternating optimization can also be achieved using a non-strict alternation / proportional alternation method (i.e., it does not require variables A and B to be updated strictly in a 1:1 ratio). For example, the hidden dimension can be fixed first, and multiple local searches can be performed on the intermediate dimension (updating A), then the intermediate dimension can be fixed again, and the hidden dimension can be updated once (updating B). That is to say, the alternating optimization described in the embodiments of this specification is not limited to a strict 1:1 rotation; it can also involve performing multiple updates on one dimension before switching to another dimension for updates. Alternatively, alternating optimization can be achieved using block coordinate descent (i.e., when there are many variables, instead of fixing / updating only "one" variable each time, the variables are divided into multiple "blocks" for batch updates to improve efficiency). For example, the channels of the intermediate dimension can be divided into 3 groups; each time, the other groups and the hidden dimension are fixed, only the first group is optimized, then the second group is optimized, and so on, finally optimizing the hidden dimension as a whole. Therefore, the alternating optimization described in the embodiments of this specification can refer to the number of channels retained in the entire dimension, or the number of channels divided into multiple sub-blocks. Alternatively, alternating optimization with a relaxation factor can be employed. This means that when updating variables, not only the optimal solution calculated in the current step is used, but also a weighted sum of the values ​​from the previous step (e.g., new value = 0.7 * current optimal solution + 0.3 * previous value) to accelerate convergence. In other words, a relaxation factor or historical state weighting mechanism can be introduced during alternating optimization to accelerate algorithm convergence. Alternatively, joint optimization can be performed on the aforementioned intermediate and hidden dimensions. This involves concatenating the number of retained channels in both dimensions into a joint vector and updating it simultaneously using algorithms such as gradient descent.

[0058] Figure 3 This is a schematic diagram illustrating the implementation process of a budget-constrained coordinate descent method as provided in an exemplary embodiment. Figure 3 As shown, multiple iterations may be required, and the specific execution process of any iteration includes the following steps: Step 301: Determine the target expert sub-models to be pruned based on the importance score.

[0059] For all E expert sub-models in the hybrid expert model, E1 target expert sub-models to be pruned are first selected through some importance scoring mechanism (such as gradient magnitude, weight norm, etc.).

[0060] For example, suppose the hybrid expert model has 8 experts. After scoring, experts 3, 5, and 7 are found to be redundant. Therefore, these 3 experts are selected as the target expert sub-model, and the corresponding baseline loss is... (For example, the accuracy is 90%).

[0061] Step 302: Try two pruning strategies in parallel.

[0062] The BCD algorithm involves two different update actions: A1 compensates by reducing the number of retained intermediate channels and increasing the number of retained hidden channels according to the parameter budget constraint. A2 compensates by reducing the number of retained hidden channels and increasing the number of retained intermediate channels according to the parameter budget constraint. The step size Δe of the iteration process refers to the adjustment amount used to adjust the number of retained channels for each expert (i.e., the dimension adjustment step size). The success counter Se is used to determine whether a series of successful pruning updates have occurred.

[0063] The parameter adjustment formulas for actions A1 and A2 are as follows: in, These are the original value of the intermediate dimension of expert e at present, and the new value of the intermediate dimension of expert e after executing action A1. These are the original values ​​of the intermediate dimensions of expert e, and the new values ​​of the intermediate dimensions of expert e after executing action A2. `dmodel` is the total dimension of the model, usually referring to the input and output dimensions of the Transformer layer. `dffn` is the baseline or reference dimension of the feedforward network (FFN). `2dffn / 3dmodel` are compensation coefficients used to increase hidden dimensions when reducing intermediate dimensions, keeping the total number of parameters roughly constant.

[0064] For any of the target expert sub-models, two different parameter configuration logics are tried simultaneously. Specifically, after executing action A1 on the left, the size of the linear layer weight matrix in the current round... Updated to For example, attempting to compress the intermediate dimension by 20% resulted in a drop in accuracy to 89% and a slight increase in loss. After executing action A2 on the right, the size of the linear layer weight matrix in the current iteration... Updated to For example, we tried compressing the middle dimension by 40%. After calculation, we found that the accuracy dropped to 85%, and the loss became significant.

[0065] Step 303: Global validity assessment.

[0066] Determine if the conditions are met: in, These are the loss values ​​(such as KL divergence) of expert e on the calibration data after executing actions A1 and A2, respectively. This is the current baseline loss value for expert e (the loss before adjustment).

[0067] This step checks whether the better-performing action (A1 or A2) is better than the original state (i.e., ...). The loss value should be good (or within the allowable loss range). Specifically, if a loss function (Loss) is used, the smaller the better; if an accuracy function (Accuracy) is used, the larger the better.

[0068] If the condition is met, it indicates that the exploration effect of a certain action is better (meaning that the current pruning attempt is effective or a better balance point has been found), and then proceed to step 304a; otherwise, it indicates that the exploration effect of both actions is not good (meaning that the pruning is too aggressive or the direction is wrong, and the model performance drops too much), and then proceed to step 304b.

[0069] Step 304a: Double the step size Δe and update the success counter Se=(Se+1)%2; then determine which action A1 or A2 has a smaller loss value and update the parameters corresponding to that action.

[0070] The step size Δe is doubled, and the success counter Se = (Se+1)%2 is updated as follows: This indicates that the exploration direction for a certain action is correct (or that pruning can continue at the current step size). In that case, we can be more aggressive and adjust the pruning amplitude (step size) of the next step to be larger, and try again to accelerate convergence. For example, if we find that compressing by 20% is perfectly fine and there is almost no loss of accuracy, we decide to "try compressing by 40% next time!" (double the step size).

[0071] Furthermore, compare action A1 and action A2 to determine which is better (i.e., which has a smaller loss): if action A1 is better (Yes), then update the parameters to A1. If action A2 is better (No), then update the parameters to A2.

[0072] In step 304b, the step size Δe is halved, and the success counter Se is updated to 0, which means the counter is reset.

[0073] In this process, the step size Δe is halved, and the success counter Se is updated to 0, as follows: Step 305: Update the access status of the current target expert sub-model and determine whether the convergence condition is met. If it is met, stop the iteration and determine the number of intermediate channels and hidden channels to be retained at this time.

[0074] Otherwise, proceed to the next iteration. That is, with the updated parameters (which may be the parameters of A1 or A2) and the new step size, return to the beginning and continue to perform the next round of pruning evaluation on the expert sub-model until the predetermined compression ratio or convergence is achieved.

[0075] To illustrate the principle of the BCD process with a simple example: it's like going downhill (finding the optimal pruning ratio). First, take a big step downhill (double the step size). If you find you haven't fallen and are closer to the finish line (Se+1), then run the next step (double the step size again). If you fall while taking a big step (Failure), go back and compare the two smaller steps you just took to see which one is more stable (compare the loss values ​​of A1 and A2). Choose the more stable step and stay there. Then, next time, only take small, incremental steps (halve the step size / fine-grained search). This method can significantly speed up the pruning search and avoid wasting time in ineffective search spaces.

[0076] Besides using the aforementioned BCD algorithm to determine the intermediate and hidden channel retention strategies, other methods can be selected based on the specific circumstances. For example, a greedy strategy based on importance scores can be employed. Instead of pre-setting a total budget for global optimization, the importance score of each channel is first calculated (e.g., based on the L1 norm and / or L2 norm of the weight magnitude, a first-order Taylor expansion approximation, or the Hessian matrix trace). Then, channels are directly sorted according to their scores, and lower-ranked channels are removed according to a pre-set fixed proportion (e.g., pruning 20% ​​at each layer) or a global threshold. This method is simple and fast, resulting in high efficiency in strategy determination. Another example is a search strategy based on reinforcement learning. This involves treating "how many intermediate channels to retain" and "how many hidden channels to retain" as the action space, and model accuracy or reconstruction error as the reward function. A controller (such as a recurrent neural network RNN ​​or Transformer) is trained to output the retention strategy, learning the optimal strategy through interaction with the environment (pruning → fine-tuning → evaluation). This approach can explore more complex nonlinear relationships and produces higher-quality strategies. For example, a strategy based on differentiable architecture search can be employed, such as introducing a set of continuously learnable mask parameters and relaxing the discrete "pruning / retaining" choices to continuous real numbers between [0,1]. Gradient descent is used to simultaneously optimize network weights and mask parameters, and after training, the parameters are truncated directly based on their magnitude (e.g., retaining those greater than 0.5). This approach can utilize GPUs to accelerate gradient computation, making it more efficient than RL, and allows for end-to-end joint optimization. Another example is a heuristic strategy based on sensitivity analysis, such as testing the impact (sensitivity) of changing intermediate and hidden dimensions on the loss, and prioritizing pruning those dimensions with the least impact on the loss (least sensitive) until the target number of parameters is reached. This approach is logically intuitive, similar to a simplified greedy algorithm, and does not require complex iterative optimization loops.

[0077] The intermediate channel retention strategy determined by the aforementioned method may include row indexes of the rows to be retained under the intermediate dimension. These row indexes are used to indicate which intermediate channels (i.e., rows to be retained) should be retained when performing channel pruning in the intermediate dimension. The hidden channel retention strategy may include column indexes of the columns to be retained under the hidden dimension. These column indexes are used to indicate which hidden channels (i.e., columns to be retained) should be retained when performing channel pruning in the hidden dimension. And / or, the intermediate channel retention strategy may also include row indexes of the rows to be pruned under the intermediate dimension. These row indexes are used to indicate which intermediate channels (i.e., rows to be pruned) should be pruned (i.e., deleted) when performing channel pruning in the intermediate dimension. The hidden channel retention strategy may include column indexes of the columns to be pruned under the hidden dimension. These column indexes are used to indicate which hidden channels (i.e., columns to be pruned) should be pruned (i.e. deleted) when performing channel pruning in the hidden dimension.

[0078] Step 1044: For the first linear layer, second linear layer and third linear layer of any target expert sub-model, perform channel pruning of the intermediate dimension according to the intermediate channel retention strategy.

[0079] Step 1046: For the first and second linear layers of any target expert sub-model, perform channel pruning of the hidden dimensions according to the hidden channel preservation strategy, while keeping the output dimension of the third linear layer unchanged.

[0080] Based on the aforementioned intermediate channel retention strategy, the intermediate dimensions of the first, second, and third linear layers (such as the aforementioned gated linear layer, upward-projected linear layer, and downward-projected linear layer) of any target expert sub-model can be pruned to reduce the number of intermediate channels in these linear layers. It is worth noting that in deep learning frameworks such as PyTorch and TensorFlow, the core entity of a linear layer at the code level includes a weight matrix and a bias vector. Therefore, the process of "pruning a linear layer" is essentially the process of pruning the rows and / or columns of the weight matrix of that linear layer. The following examples use gated linear layers, upward-projected linear layers, and downward-projected linear layers as examples: In one embodiment, when performing intermediate channel pruning, the weight matrices of the gated linear layer, the upprojected linear layer, and the downprojected linear layer of any target expert sub-model can be obtained first. Then, based on the intermediate channel retention strategy, the row indices of the rows to be retained in the weight matrices of the gated linear layer and the upprojected linear layer, and the column indices of the columns to be retained in the weight matrix of the downprojected linear layer are determined respectively. Subsequently, according to the row indices and the column indices, the weight matrices of the gated linear layer, the upprojected linear layer, and the downprojected linear layer are pruned, that is, each row and column to be pruned is removed from the weight matrices, while the corresponding rows and columns to be retained are retained (i.e., not pruned). As mentioned above, for the gated linear layer and the upprojected linear layer, pruning according to the row index is actually reducing the output dimension of both, that is, compressing the number of output channels of both; while for the downprojected linear layer, pruning according to the row index is actually reducing the input dimension of the linear layer, that is, compressing the number of input channels of the linear layer.

[0081] The intermediate dimension pruning method in this embodiment enables seamless connection between layers: because the outputs of the gated layer and the upper projection layer are the inputs of the lower projection layer, the strict correspondence between "row pruning" and "column pruning" ensures that the feature vector length of the previous layer's output is completely consistent with the received length of the next layer's input (i.e., both become dffn'). In other words, this pruning method is determined by the connection relationship of the three linear layers: the outputs of the gated linear layer and the upper projection linear layer are connected to the input of the lower projection linear layer, so pruning is required in the above manner to ensure that the input of the pruned lower projection linear layer can be aligned with the outputs of the gated linear layer and the upper projection linear layer. Moreover, it avoids matrix multiplication errors caused by dimension mismatch, thereby ensuring the validity of matrix operations and allowing the pruned model to run directly without an additional adaptation layer. In addition, because it directly reduces the most time-consuming wide matrix multiplication operation inside FFN (from dmodel×dffn to dmodel×dffn'), it can accurately compress the computational load and significantly reduce inference latency.

[0082] In another embodiment, when performing hidden dimension channel pruning, the weight matrices of the gated linear layer and the upprojected linear layer of any target expert sub-model can be obtained first. Then, based on the hidden channel retention strategy, the column indices of the columns to be retained in the weight matrices of the gated linear layer and the upprojected linear layer are determined. Subsequently, the weight matrices of the gated linear layer and the upprojected linear layer are pruned according to the column indices. As mentioned earlier, pruning the gated linear layer and the upprojected linear layer according to the column indices effectively reduces their input dimensions, i.e., compresses the number of input channels. This hidden dimension pruning method reduces the model's dependence on input features, thereby reducing input redundancy. If there is a large amount of redundant information at the input (e.g., some embedding dimensions contribute little to the task), this pruning method can effectively remove noise. Moreover, because the width of the weight matrix is ​​reduced, the number of parameters can be further compressed, thereby reducing storage requirements during deployment and inference.

[0083] like Figure 2 As shown, the horizontal scissors icon represents the rows to be clipped from the weight matrix, and the vertical scissors icon represents the columns to be clipped from the weight matrix. After clipping, the remaining rows and columns to be retained together form a new linear layer. For gated linear layers and upprojected linear layers, the size before clipping is dffn×dmodel, and the size after clipping is dffn′×dmodel′.

[0084] It should be noted that when performing channel pruning for intermediate and hidden dimensions in the manner described above, it is usually also necessary to adjust the upstream modules of each expert (such as...). Figure 2 The outputs of the normalization layer (LayerNorm), attention layer (Attention), residual connections, and normalization layers (Add&Norm, etc.) shown are also pruned accordingly, or a projection layer is added at the entry point of the MoE module to adapt the dimensions, thereby achieving end-to-end feature narrowing transmission. Furthermore, although steps 1044 and 1046 are written as two independent steps, in actual pruning, they can be completed simultaneously rather than as two separate steps; in other words, these two steps do not have a necessary order and can be executed sequentially, in parallel, or even simultaneously. This specification does not impose any limitations on this. For example, the linear layers can be processed in parallel or sequentially: for any linear layer in the gated linear layer and the up-projected linear layer, the channel pruning of the intermediate and hidden dimensions can be completed simultaneously, i.e., the pruning operations of the rows and columns to be pruned are completed at once; while for the down-projected linear layer, only the channel pruning of the intermediate dimensions needs to be performed, without needing to prune the hidden dimensions, to ensure that the output dimension of the linear layer remains unchanged.

[0085] Through the aforementioned "channel pruning" process, some channels in the linear layer can be completely pruned, that is, each channel (i.e., the row and column to be pruned) is completely removed from the weight matrix of the linear layer as a basic unit. After the above pruning of each target expert sub-model is completed, the pruned expert sub-model composed of the remaining channels (i.e., the rows and columns to be retained) can be obtained. At this time, each pruned expert sub-model (such as...) Figure 2 The E1 sub-models shown) and the unpruned expert sub-models mentioned above (such as...) Figure 2 The E2 shared expert sub-models shown together constitute the pruned expert model. It can be seen that the pruning process of the hybrid expert model is actually the process of modifying the shape of the weight matrix in the model according to instructions and discarding useless channels / parameters—the remaining useful channels / parameters constitute the pruned expert model.

[0086] As can be seen from the above embodiments, the novel structured pruning scheme proposed in this specification introduces a "hidden dimension channel preservation strategy" to prune the hidden dimensions of the gated linear layer and the upprojected linear layer (e.g., pruning their input ends). This expands the compression dimension from a single "intermediate dimension" to a two-dimensional collaborative compression of "intermediate dimension + hidden dimension," enabling in-depth mining and removal of redundant features at the input interface of the MoE model, thereby significantly improving the model's parameter compression efficiency. In fact, because the pruning of the hidden dimension reduces the input width of the expert sub-model, combined with the pruning of the intermediate dimension, a compression rate far exceeding that of traditional single-dimensional schemes can be achieved without compromising the model's core reasoning capabilities. Furthermore, this scheme first identifies the target expert sub-models to be pruned, then determines the intermediate channel retention strategy and hidden channel retention strategy for each target expert sub-model, and performs targeted structured pruning accordingly. This "expert-specific" adaptive channel retention mechanism allows the pruning operation to accurately match the unique redundancy distribution within different expert sub-models, avoiding the accidental deletion of key features that might be caused by a globally uniform pruning ratio, thus effectively ensuring the inference performance of the pruned model. In addition, by "keeping the output dimension of the downprojected linear layer unchanged," it ensures that the pruned expert sub-model can still be aligned with the global residual connection of the MoE model at the output, without introducing any additional alignment projection or recovery operators. Therefore, it not only achieves the compression of hidden dimensions but also maintains the regularity of the overall model topology, ensuring that the pruned model can seamlessly interface with existing intensive computing hardware and avoiding a reduction in deployment efficiency due to compression.

[0087] In one embodiment, after pruning the hybrid expert model (i.e., performing structured pruning on each target expert sub-model separately), the pruning effect of the pruned expert model can be verified based on the Attention Relation Transfer Distillation (RTD) algorithm. For example, after pruning each target expert sub-model, for the pruned expert model and its corresponding standard expert model, the same verification data can be input into the pruned expert model and the standard expert model respectively, and the pruned feature map output by the pruned expert model and the standard feature map output by the standard expert model can be obtained. The pruned feature map contains the sample feature vectors corresponding to the verification data in the pruned expert model, and the standard feature map contains the sample feature vectors corresponding to the verification data in the standard expert model. Then, the structural relationship difference between the pruned feature map and the standard feature map is determined, and the model parameters of the pruned expert model are updated based on the structural relationship difference.

[0088] In this embodiment, the "feature map" is not a two-dimensional image matrix in traditional computer vision, but rather a set of feature representations generated by the model in the intermediate or output layers when processing the input sequence. For example, when a sentence (composed of multiple tokens) is input into the model, each token is mapped to a high-dimensional vector. The "feature map" is a matrix formed by arranging the sample feature vectors corresponding to these tokens in sequence, containing the model's understanding and semantic representation of each word in the sentence. The standard feature map mentioned above is extracted from the standard expert model, while the pruned feature map is extracted from the pruned expert model.

[0089] For example, let's assume we want to prune a hybrid expert model of a BERT (Bidirectional Encoder Representation from Transformers) model, with the validation data being the sentence: "Artificial intelligence changes the world." The word segmentation result of this sentence is ["artificial", "intelligent", "change", "world"], showing that the token sequence contains a total of 4 tokens. Next, we input this token sequence into a standard expert model and extract the feature vector of a certain layer of this model as a standard feature map. Let's assume the hidden dimension of this expert is 768, which gives us a 4×768 matrix, denoted as Fstd. Each row in this matrix represents the semantic state of a token (such as "artificial") in the original model. Alternatively, we can input the token sequence into the pruned expert model (assuming the hidden dimension of the expert in this model is compressed to 512 dimensions), and extract the corresponding output to obtain a 4×512 matrix, denoted as Fpruned. Then, we can calculate the structural relationship difference, such as directly fitting the 4×512 to the 4×768 (because the dimensions are different, direct subtraction is not possible). Further relational transfer is performed, such as the similarity matrix (relationship graph) between tokens in Fstd: for example, the cosine similarity between "artificial" and "intelligent" is 0.9, and between "artificial" and "world" it is 0.2, which constitutes the internal topology of the standard model. The similarity matrix between tokens in Fpruned is also calculated to verify whether the similarity between "artificial" and "intelligent" in the pruned expert model is still close to 0.9. Then, parameters are updated, such as constructing a loss function (Loss) to penalize the difference between the two similarity matrices (MSE or KL divergence). Finally, the weights of the pruned expert model are updated through backpropagation, so that although the dimensionality is reduced, the relative semantic relationships between tokens remain consistent with the standard model.

[0090] As can be seen, this embodiment achieves cross-dimensional knowledge distillation by introducing structural relationship differences as a supervision signal, solving the problem that the pruned model and the standard model cannot be directly aligned due to feature dimension mismatch. Secondly, by using the RTD algorithm to force the pruned expert model to mimic the relative topological structure between tokens in the standard expert model, it effectively preserves long-distance dependencies and contextual semantic logic. This constraint of intermediate layer feature relationships can more finely guide model parameter updates, significantly alleviating the accuracy decline problem caused by channel pruning. Traditional model distillation involves having "Token A of the pruned expert model" mimic "Token A of the standard expert model" (point-to-point learning); while the RTD algorithm used in this embodiment calculates the "similarity between Token A and Token B" in the pruned expert model and then mimics the "similarity between Token A and Token B" in the standard expert model, achieving model distillation and knowledge transfer from the perspective of the model's internal structure, thereby avoiding the problem of decreased inference accuracy caused by parameter mismatch in hybrid expert models. Furthermore, this method uses validation data to dynamically calibrate the model, enhancing the robustness and generalization ability of the pruned model on specific tasks.

[0091] The standard expert model can be the hybrid expert model before pruning, in which case the models before and after pruning are compared. Alternatively, the hybrid expert model can be a student model obtained by distillation training based on the teacher model, in which case the standard expert model can be the teacher model, that is, the pruned student model can be compared with the teacher model used to distill and obtain the original student model (i.e., the hybrid expert model).

[0092] In one embodiment, when determining the structural relationship difference between the pruned feature map and the standard feature map, a first similarity matrix between the feature vectors of each sample in the pruned feature map and a second similarity matrix between the feature vectors of each sample in the standard feature map can be calculated first. Then, the mean squared error (MSE) between the first similarity matrix and the second similarity matrix is ​​calculated as the structural relationship difference between the pruned feature map and the standard feature map. The first similarity matrix and the second similarity matrix can be matrices of the same order. For example, both can be first-order similarity matrices (i.e., 1×N or N×1 one-dimensional vectors); or both can be second-order similarity matrices (i.e., M×N or N×M two-dimensional matrices), and these two vectors can characterize the internal topological structure of the feature map. By calculating the MSE of the two second-order matrices, it can be ensured that the pruned feature map is not only close to the standard feature map in absolute value, but also that the two-dimensional topological structure of the relative relationships between samples remains unchanged.

[0093] This embodiment solves the problem of mismatched feature map dimensions before and after pruning, which prevents direct alignment, by transforming absolute high-dimensional feature vectors into relative relational topologies, thus achieving effective cross-dimensional knowledge distillation. Simultaneously, by constructing a similarity matrix and calculating the mean squared error (MSE), it forces the pruned expert model to mimic the word-to-word relationships in the standard expert model, accurately preserving the global semantic structure and long-distance dependencies. Furthermore, the calculation of the similarity matrix and MSE is very lightweight, and the MSE is smooth and differentiable, facilitating gradient backpropagation and enabling stable and rapid updating of model parameters. Finally, this holistic constraint method based on "relationship to relation" is smoother than traditional point-to-point fitting and has a higher tolerance for small perturbations or noise in individual tokens, thereby significantly enhancing the model's robustness and generalization performance.

[0094] For example, let's take a hybrid expert model as the student model and a standard expert model as the corresponding teacher model. We can first obtain the attention components, such as extracting the activation values ​​of the query (Q), key (K), and value (V) matrices after linear projection from the L-th layer and a-th attention head of the teacher model (T) and student model (S). Then, we construct the relationship matrix, such as calculating the relationship matrix between tokens based on the Q, K, and V vectors mentioned above using the following formula: in It is the calculated attention distribution matrix, which is usually used for the target distribution in knowledge distillation. It is the attention matrix of the teacher model at the Lth layer and the ath head. This refers to the L2 norm (Euclidean norm), which can refer to some kind of normalization or projection operation on the attention matrix (depending on the context, it could also refer to some kind of magnitude calculation of the matrix itself). `dhead` is the dimension of each attention head, used to scale the dot product result and prevent gradient vanishing. `Softmax` is the normalization function used to convert numerical values ​​into a probability distribution. As you can see, this formula transforms absolute activation values ​​into relative inter-token relationships by calculating the L2 norm product of vectors and performing Softmax normalization.

[0095] Next, the distillation loss can be calculated. The optimization objective can be the following formula: minimizing the KL divergence (DKL) of the relationship matrix between teachers and students. in This is the total loss value of RTD. Nhead is the total number of attention heads. |Φ| is the size of the selected layers set Φ, i.e., the number of layers involved in the computation. L∈Φ means traversing the selected layer indices L. A∈{Q,K,V} means traversing the three attention mechanism components: Query (Q), Key (K), and Value (V). DKL(·||·) refers to the KL divergence, used to measure the difference between two probability distributions. and These represent the attention distributions of the teacher model and the student model at layer L, component A, and head a, respectively.

[0096] In another embodiment, when updating the model parameters of the pruned expert model based on the structural relationship differences, the structural relationship differences can first be used as the distillation loss function of the pruned expert model to calculate the gradient of the weights of each network layer in the pruned expert model; then, the model parameters of the pruned expert model are iteratively updated according to the gradients to make the feature relationships of the pruned expert model approximate the feature relationships of the standard expert model. Unlike traditional point-to-point feature alignment, this embodiment directly transforms the structural relationship differences into a distillation loss function, enabling the pruned expert model to accurately inherit the high-order semantic topology of the standard model even with reduced feature dimensions, achieving lossless cross-dimensional knowledge transfer. Furthermore, gradient calculation provides clear mathematical guidance for the iterative update of model parameters, ensuring that parameter adjustments always move towards the goal of "approximating the feature relationships of the standard model," thereby guaranteeing stable convergence of the optimization and avoiding a precipitous performance drop caused by blind pruning. Moreover, since the optimization objective of this scheme focuses on the relative relationship between features rather than absolute values, it can effectively prevent the model from losing long-distance dependencies and contextual associations during compression, thereby deeply preserving the model's inherent logic and reasoning ability, and maintaining the intelligence level and generalization performance of the original model as much as possible.

[0097] Besides feature map-based relation distillation methods, multi-dimensional evaluation techniques can be combined to comprehensively verify the model pruning effect. For example, downstream task performance evaluation can serve as the most direct verification standard. This can be achieved by comparing accuracy, F1 score, or perplexity on standard test sets such as GLUE (General Language Understanding Evaluation) or SQuAD (Stanford Question Answering Dataset) to intuitively measure the success of pruning. Furthermore, inference latency and throughput tests can be performed on the pruned expert model. Measuring inference time and throughput on actual hardware such as GPUs or mobile devices verifies whether pruning brings real computational acceleration, rather than merely a reduction in parameters. Additionally, to delve into the extent of damage to the model's original capabilities, probe task analysis can be employed. Linguistic probing tasks such as part-of-speech tagging can detect the model's retention of grammatical and semantic knowledge. Simultaneously, sensitivity analysis can be combined to accurately pinpoint pruning bottlenecks and verify the rationality of the strategy by restoring parameters layer by layer and observing the performance recovery curve. For example, activation value distribution visualization techniques such as t-SNE (t-Distributed Stochastic Neighbor Embedding) can be used to observe the clustering of tokens in the model feature space before and after pruning. If the semantics that should have been clustered become discrete or disordered, it indicates that pruning has destroyed the original semantic space structure.

[0098] In one embodiment, the hybrid expert model may further include a routing network (such as...) Figure 2The Router shown is used to determine the activation weights of each expert sub-model in the hybrid expert model based on the input data. Furthermore, as described in the aforementioned intermediate channel retention strategy, for any target expert sub-model, when pruning the intermediate channels of the first and second linear layers of that sub-model, all intermediate channels of that linear layer may be pruned, effectively removing the target expert sub-model from the hybrid expert model. In this case, the routing network of the pruned expert model naturally does not need to distribute tokens to the target expert sub-model during inference. To achieve this effect, the following processing can be performed: for example, if at least one target expert sub-model's intermediate channels are pruned during pruning, the weight matrix of the routing network can be obtained, and the rows or columns corresponding to the target expert sub-model in the weight matrix can be deleted to ensure that the output dimension of the routing network is consistent with the number of effective expert sub-models in the pruned expert model. Suppose the hybrid expert model contains three expert sub-models (experts A, B, and C), and the weight matrix of the routing network is 3×N-dimensional (each row corresponds to the activation weight of one expert). If all intermediate channels of expert B are pruned during pruning (i.e., expert B is completely removed), the weight matrix can be obtained by deleting the rows corresponding to expert B, resulting in a new 2×N-dimensional weight matrix. In this case, the routing network only outputs the activation weights of experts A and C, with the dimension matching the number of remaining effective experts, thus avoiding invalid calculations for the removed expert B.

[0099] This scheme achieves dynamic matching between the output dimension of the routing network and the number of effective experts after pruning by synchronously pruning the rows or columns in the routing network weight matrix corresponding to the fully pruned experts. This eliminates computational redundancy for invalid experts, improves inference efficiency, ensures the accuracy of route allocation, avoids model running errors caused by dimension mismatch, and maintains the structural integrity and functional consistency of the hybrid expert model.

[0100] In one embodiment, the target channel indices to be retained in the hidden dimension of the gated linear layer and the upprojected linear layer can also be determined according to the hidden channel retention strategy; and a convergence operator can be configured at the input of the feedforward network of any target expert sub-model. Specifically, during the inference phase of the pruned expert model, the convergence operator (such as...) Figure 2The Gather operator (shown) is used to select channels in the input data of the feedforward network of any target expert sub-model according to the target channel index, so as to align the dimensions of the input data to the hidden dimensions of the pruned gated linear layer and the upprojected linear layer. In the hybrid expert model, the Gather operator is used for dimension alignment and data filtering. It is located between the output of the preceding module and the gated linear layer and the upprojected linear layer of the current expert sub-model, and its core function is to solve the problem of mismatch between the dimensions of the input data and the dimensions of the pruned model weights. Since the activation values ​​passed by the preceding module retain the full dimensions, while the hidden dimensions of the gated linear layer and the upprojected linear layer have been pruned, retaining only some channels, the Gather operator can accurately extract the corresponding feature channels from the full input based on the retained channel index shared by these two linear layers, discarding redundant data. Through this operation, the Gather operator can compress and align the dimensions of the input data to the actual hidden dimensions of the pruned gated linear layer and the upprojected linear layer, thereby ensuring the correct execution of subsequent matrix multiplication and efficient inference.

[0101] Assume the original dimension of the hidden layer of the feedforward network (FFN) of the expert sub-model in the hybrid expert model is 4096. Based on a hidden channel retention strategy (e.g., L1 norm-based selection), the 2048 channels with the highest importance in the hidden dimension are determined, resulting in a target channel index set I = {i1, i2, ..., i2048}. During the inference stage of the pruned expert sub-model, a convergence operator is deployed at the input of the FFN. When input data (4096 dimension) enters the expert, the convergence operator directly extracts the corresponding 2048 feature values ​​from the input data according to the index set I, constructing a new input tensor of dimension 2048. This tensor is then input to the pruned gated linear layer and the upprojected linear layer (whose hidden dimension has also been adjusted to 2048), thus achieving seamless dimension alignment. This scheme, by configuring a convergence operator at the input of the feedforward network of the expert sub-model, achieves dynamic channel selection and dimension alignment based on the target channel index, effectively solving the problem of input data and model parameter dimension mismatch caused by hidden dimension reduction after pruning. This not only avoids redundant computation and memory waste caused by full data input, ensuring the correctness and efficiency of the pruned model's inference, but also further improves the feature processing efficiency and running speed of the pruned expert model on specific tasks by extracting key feature channels as needed.

[0102] The preceding embodiments disclosed a structured pruning scheme for hybrid expert models. After pruning, the pruned expert model can be deployed in the corresponding software and hardware runtime environment, and the model can be used to provide online or offline inference services. Figure 4This is a flowchart illustrating an exemplary embodiment of a reasoning method based on a hybrid expert model, which can be applied to a reasoning server. For example... Figure 2 As shown, the method includes steps 402-404.

[0103] Step 402: Obtain the target prompt words to be processed.

[0104] The inference server can provide callable interfaces (inference service APIs) to allow users to initiate inference requests through the corresponding inference client. Upon receiving the request, the inference server can parse the target prompt words (such as a piece of natural language text) from the request and use them as input data for the target hybrid expert model.

[0105] For example, a user inputs a natural language text question, "Please briefly describe the main plot of the novel 'xxx' and analyze its literary value," through an inference client (app, HTTP page, mini-program, etc.). The inference client can then use this question as a target prompt to initiate an inference request to the inference server. The server's preprocessing module (Tokenizer) can segment this text into a token sequence (e.g., [t1, t2, ..., tn]) and convert it into a vector format recognizable by the model, ready for input to the target hybrid expert model.

[0106] Step 404: Input the target prompt word into the target hybrid expert model and obtain the inference result output by the target hybrid expert model. This target hybrid expert model is the pruned expert model obtained by performing structured pruning on the hybrid expert model as described in the previous embodiment.

[0107] After the aforementioned structured pruning is completed, the target hybrid expert model contains at least one expert sub-model (such as an unpruned shared expert sub-model, a target expert sub-model with partially pruned weights, etc.). Upon receiving the inference request, the target hybrid expert model first performs routing distribution, that is, the token sequence of the aforementioned target prompt words first enters the routing network, and the Router determines which expert sub-models (hereinafter referred to as experts) in the model to distribute these tokens to for processing based on the content of the tokens.

[0108] For each selected expert, the data is processed through a convergence operator followed by a gated linear layer and an upprojective linear layer. This convergence operator extracts specific dimensional features from the input vector based on the "target channel index" preserved by a pruning strategy, discarding redundant dimensions. Subsequently, the data processed by the gated linear layer and the upprojective linear layer is processed by the downprojective linear layer before being output. The outputs of all activated experts are then aggregated (e.g., weighted summation) to finally output the text response (i.e., the answer generated by the model for the aforementioned question).

[0109] Understandably, in the construction phase of the target hybrid expert model, redundant channels are removed through joint pruning using "intermediate dimensions + hidden dimensions," and input features are directly filtered using convergence operators during the inference phase. This significantly reduces the amount of matrix multiplication operations, thereby significantly reducing inference latency and memory usage on the same hardware and improving concurrent processing capabilities. Secondly, with the help of RTD technology, the compressed model can effectively mimic the attention relationships and semantic topology of the full-size teacher model, ensuring that the output inference results maintain high logic, coherence, and knowledge accuracy even with narrowed internal dimensions of the experts, effectively avoiding the performance degradation problem caused by traditional pruning. In addition, the method maintains the invariance of the output dimension of the expert downprojection linear layer in its architectural design and avoids additional 2D indexing operations, enabling the pruned model output to align residual connections. The server can be deployed directly using standard intensive computing libraries without modifying the underlying inference framework, effectively lowering the threshold for engineering implementation.

[0110] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 5 At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into memory 508 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0111] refer to Figure 6 The structured pruning device using hybrid expert models can be applied to, for example... Figure 5 The device shown is used to implement the technical solution of this specification. The device includes: The model determination unit 601 is used to determine the target expert sub-model to be pruned among multiple expert sub-models contained in the hybrid expert model. The feedforward network of each expert sub-model in the multiple expert sub-models includes a first linear layer, a second linear layer and a third linear layer. The structural dimensions of the feedforward network include intermediate dimensions and hidden dimensions. The model pruning unit 602 is used to prune the target expert sub-model; the following sub-units of the model pruning unit 602 are used to prune any target expert sub-model, wherein: The strategy determination subunit 6021 is used to determine the intermediate channel retention strategy of any target expert submodel in the intermediate dimension and the hidden channel retention strategy in the hidden dimension. The intermediate pruning subunit 6022 is used to prune the intermediate dimensions of the first linear layer, the second linear layer and the third linear layer of any target expert submodel according to the intermediate channel retention strategy. The hidden pruning subunit 6023 is used to perform hidden dimension pruning on the first and second linear layers of any target expert submodel according to the hidden channel retention strategy, while keeping the output dimension of the third linear layer unchanged.

[0112] Optionally, the model determining unit 601 is specifically used for: Calculate the importance score of each expert sub-model among the multiple expert sub-models contained in the hybrid expert model; The expert sub-models whose importance scores are below a preset threshold, or the preset number of expert sub-models with the lowest importance scores, are identified as target expert sub-models to be pruned.

[0113] Optionally, the strategy determining subunit 6021 is specifically used for: Obtain the total parameter budget of any target expert sub-model, whereby the total parameter budget is used to characterize the total number of parameters of the target expert sub-model in the intermediate and hidden dimensions after pruning. With the goal of minimizing model reconstruction error, under the constraint of satisfying the total budget of the parameters, the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension are alternately optimized until the convergence condition is met; Based on the number of intermediate channels retained and the number of hidden channels retained after convergence, corresponding intermediate channel retention strategies and hidden channel retention strategies are generated respectively.

[0114] Optionally, the strategy determining subunit 6021 is specifically used for: Initialize the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension; Under the constraint of satisfying the total budget of the parameters, the search is performed in an alternating iterative manner between the number of intermediate channels retained and the number of hidden channels retained until the convergence condition is met; wherein, in each search, the number of channels retained is fixed, either the number of intermediate channels retained or the number of hidden channels retained, and the number of channels retained is updated with minimizing the model reconstruction error as the objective function.

[0115] Optionally, the intermediate trimming subunit 6022 is specifically used for: Obtain the weight matrices of the first linear layer, the second linear layer, and the third linear layer of any target expert sub-model; Based on the intermediate channel retention strategy, the row indices of the rows to be retained in the weight matrices of the first linear layer and the second linear layer, and the column indices of the columns to be retained in the weight matrix of the third linear layer are determined respectively. The weight matrices of the first linear layer, the second linear layer, and the third linear layer are pruned according to the row index and the column index, respectively.

[0116] Optionally, the hidden clipping subunit 6023 is specifically used for: Obtain the weight matrices of the first and second linear layers of any target expert sub-model; Based on the hidden channel retention strategy, the column indices of the columns to be retained in the weight matrices of the first linear layer and the second linear layer are determined; Based on the column index, the weight matrices of the first linear layer and the second linear layer are clipped respectively.

[0117] Optionally, the device further includes a model verification unit 603, used for: After pruning each target expert sub-model, the same validation data is input into the pruned expert model and the corresponding standard expert model, respectively, and the pruned feature map output by the pruned expert model and the standard feature map output by the standard expert model are obtained. The pruned feature map contains the sample feature vector corresponding to the validation data in the pruned expert model, and the standard feature map contains the sample feature vector corresponding to the validation data in the standard expert model. The structural relationship differences between the pruned feature map and the standard feature map are determined, and the model parameters of the pruned expert model are updated based on the structural relationship differences.

[0118] Optionally, the model verification unit 603 is specifically used for: Calculate the first similarity matrix between the feature vectors of each sample in the pruned feature map, and the second similarity matrix between the feature vectors of each sample in the standard feature map; The mean squared error between the first similarity matrix and the second similarity matrix is ​​calculated as the structural relationship difference between the pruned feature map and the standard feature map.

[0119] Optionally, the model verification unit 603 is specifically used for: The structural relationship differences are used as the distillation loss function of the pruned expert model to calculate the gradient of the weights of each network layer in the pruned expert model. The model parameters of the pruned expert model are iteratively updated according to the gradient so that the feature relationships of the pruned expert model approximate the feature relationships of the standard expert model.

[0120] Optionally, The standard expert model is the hybrid expert model before pruning; or... In the case where the hybrid expert model is a student model obtained by distillation training based on the teacher model, the standard expert model is the teacher model.

[0121] Optionally, the hybrid expert model further includes a routing network, which is used to determine the activation weights of each expert sub-model in the hybrid expert model based on the input data; the device further includes a routing adaptation unit 604, used for: If all intermediate channels of at least one target expert sub-model are pruned during the pruning of the target expert sub-model, then the weight matrix of the routing network is obtained, and the row or column corresponding to the target expert sub-model in the weight matrix is ​​deleted, so that the output dimension of the routing network is consistent with the number of effective expert sub-models in the pruned expert model.

[0122] Optionally, it also includes a convergence configuration unit 605 for: The target channel indices to be retained in the hidden dimension are determined according to the hidden channel retention strategy of the first linear layer and the second linear layer. A convergence operator is configured at the input end of the feedforward network of any target expert sub-model. During the inference stage of the pruned expert model, the convergence operator is used to select channels for the input data input to the feedforward network of any target expert sub-model according to the target channel index, so as to align the dimensions of the input data to the hidden dimensions of the pruned first linear layer and the second linear layer.

[0123] refer to Figure 7 Inference devices based on hybrid expert models can be applied to, for example... Figure 5 The device shown is used to implement the technical solution of this specification. The device includes: The prompt word acquisition unit 701 is used to acquire the target prompt words to be processed; The model invocation unit 702 is used to input the target prompt word into the target hybrid expert model and obtain the inference result output by the target hybrid expert model; wherein, the target hybrid expert model is obtained by performing structured pruning on the hybrid expert model as described in any of the preceding embodiments.

[0124] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0125] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0126] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

Claims

1. A structured pruning method using a hybrid expert model, characterized in that, include: In a hybrid expert model, a target expert submodel to be pruned is determined from among multiple expert submodels. The feedforward network of each expert submodel includes a first linear layer, a second linear layer, and a third linear layer. The structural dimensions of the feedforward network include intermediate dimensions and hidden dimensions. Prune the target expert sub-model; The pruning of any target expert sub-model includes: Determine the intermediate channel retention strategy of any target expert sub-model in the intermediate dimension and the hidden channel retention strategy in the hidden dimension; For the first linear layer, second linear layer, and third linear layer of any target expert sub-model, the intermediate dimension channels are pruned according to the intermediate channel retention strategy. For the first and second linear layers of any target expert sub-model, the hidden dimensions are pruned according to the hidden channel retention strategy, while keeping the output dimension of the third linear layer unchanged.

2. The method according to claim 1, characterized in that, The step of determining the target expert sub-model to be pruned among multiple expert sub-models included in the hybrid expert model includes: Calculate the importance score of each expert sub-model among the multiple expert sub-models contained in the hybrid expert model; The expert sub-models whose importance scores are below a preset threshold, or the preset number of expert sub-models with the lowest importance scores, are identified as target expert sub-models to be pruned.

3. The method according to claim 1, characterized in that, Determining the intermediate channel retention strategy for any target expert sub-model in the intermediate dimension and the hidden channel retention strategy in the hidden dimension includes: Obtain the total parameter budget of any target expert sub-model, whereby the total parameter budget is used to characterize the total number of parameters of the target expert sub-model in the intermediate and hidden dimensions after pruning. With the goal of minimizing model reconstruction error, under the constraint of satisfying the total budget of the parameters, the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension are alternately optimized until the convergence condition is met; Based on the number of intermediate channels retained and the number of hidden channels retained after convergence, corresponding intermediate channel retention strategies and hidden channel retention strategies are generated respectively.

4. The method according to claim 3, characterized in that, The objective of minimizing model reconstruction error, under the constraint of satisfying the total parameter budget, involves alternately optimizing the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension until the convergence condition is met, including: Initialize the number of intermediate channels retained in the intermediate dimension and the number of hidden channels retained in the hidden dimension; Under the constraint of satisfying the total budget of the parameters, the search is performed in an alternating iterative manner between the number of intermediate channels retained and the number of hidden channels retained until the convergence condition is met; wherein, in each search, the number of channels retained is fixed, either the number of intermediate channels retained or the number of hidden channels retained, and the number of channels retained is updated with minimizing the model reconstruction error as the objective function.

5. The method according to claim 1, characterized in that, The process of pruning intermediate dimensions for the first, second, and third linear layers of any target expert sub-model, according to the intermediate channel retention strategy, includes: Obtain the weight matrices of the first linear layer, the second linear layer, and the third linear layer of any target expert sub-model; Based on the intermediate channel retention strategy, the row indices of the rows to be retained in the weight matrices of the first linear layer and the second linear layer, and the column indices of the columns to be retained in the weight matrix of the third linear layer are determined respectively. The weight matrices of the first linear layer, the second linear layer, and the third linear layer are pruned according to the row index and the column index, respectively.

6. The method according to claim 1, characterized in that, The process of pruning the hidden dimensions of the first and second linear layers of any target expert sub-model according to the hidden channel preservation strategy includes: Obtain the weight matrices of the first and second linear layers of any target expert sub-model; Based on the hidden channel retention strategy, the column indices of the columns to be retained in the weight matrices of the first linear layer and the second linear layer are determined; Based on the column index, the weight matrices of the first linear layer and the second linear layer are clipped respectively.

7. The method according to claim 1, characterized in that, The method further includes: After pruning each target expert sub-model, the same validation data is input into the pruned expert model and the corresponding standard expert model, respectively, and the pruned feature map output by the pruned expert model and the standard feature map output by the standard expert model are obtained. The pruned feature map contains the sample feature vector corresponding to the validation data in the pruned expert model, and the standard feature map contains the sample feature vector corresponding to the validation data in the standard expert model. The structural relationship differences between the pruned feature map and the standard feature map are determined, and the model parameters of the pruned expert model are updated based on the structural relationship differences.

8. The method according to claim 7, characterized in that, Determining the structural relationship differences between the pruned feature map and the standard feature map includes: Calculate the first similarity matrix between the feature vectors of each sample in the pruned feature map, and the second similarity matrix between the feature vectors of each sample in the standard feature map; The mean squared error between the first similarity matrix and the second similarity matrix is ​​calculated as the structural relationship difference between the pruned feature map and the standard feature map.

9. The method according to claim 7, characterized in that, The process of updating the model parameters of the pruned expert model based on the structural relationship differences includes: The structural relationship differences are used as the distillation loss function of the pruned expert model to calculate the gradient of the weights of each network layer in the pruned expert model. The model parameters of the pruned expert model are iteratively updated according to the gradient so that the feature relationships of the pruned expert model approximate the feature relationships of the standard expert model.

10. The method according to claim 7, characterized in that, The standard expert model is the hybrid expert model before pruning; or... In the case where the hybrid expert model is a student model obtained by distillation training based on the teacher model, the standard expert model is the teacher model.

11. The method according to claim 1, characterized in that, The hybrid expert model further includes a routing network, which is used to determine the activation weights of each expert sub-model in the hybrid expert model based on the input data; the method further includes: If all intermediate channels of at least one target expert sub-model are pruned during the pruning of the target expert sub-model, then the weight matrix of the routing network is obtained, and the row or column corresponding to the target expert sub-model in the weight matrix is ​​deleted, so that the output dimension of the routing network is consistent with the number of effective expert sub-models in the pruned expert model.

12. The method according to claim 1, characterized in that, Also includes: The target channel indices to be retained in the hidden dimension are determined according to the hidden channel retention strategy of the first linear layer and the second linear layer. A convergence operator is configured at the input end of the feedforward network of any target expert sub-model. During the inference stage of the pruned expert model, the convergence operator is used to select channels for the input data input to the feedforward network of any target expert sub-model according to the target channel index, so as to align the dimensions of the input data to the hidden dimensions of the pruned first linear layer and the second linear layer.

13. The method according to any one of claims 1-12, characterized in that, The first linear layer, the second linear layer, and the third linear layer are respectively a gated linear layer, an upward-projected linear layer, and a downward-projected linear layer.

14. A reasoning method based on a hybrid expert model, characterized in that, include: Obtain the target prompt words to be processed; The target prompt word is input into the target hybrid expert model, and the reasoning result output by the target hybrid expert model is obtained; wherein, the target hybrid expert model is obtained by performing structured pruning on the hybrid expert model using the method described in any one of claims 1-12.

15. An electronic device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-14 by executing the executable instructions.

16. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-14.

17. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-14.

Images