Model processing methods, apparatus, equipment, storage media, and program products

Abstract

This application discloses a model processing method, apparatus, device, storage medium, and program product, relating to the field of artificial intelligence technology. In the method of this application, singular value decomposition is performed on the network weight matrix of a hybrid expert model, and columns with singular values ​​less than a threshold are selected from the resulting right singular vector matrix as the null basis matrix. Thus, the null basis matrix can characterize the activation mode directions that contribute less to the activation of the expert network. Furthermore, a sub-network weight matrix is ​​constructed based on the columns of the null basis matrix and the network weight matrix. In this way, the proximity of the linear correlation between each column of the network weight matrix and the null basis matrix can be determined based on the maximum and minimum singular values ​​of the sub-network weight matrix, thereby determining the ineffective feature sensitivity of each expert network, and pruning the network parameters of the hybrid expert model based on the ineffective feature sensitivity. This reduces the number of model parameters.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to model processing methods, apparatus, devices, storage media and program products. Background Technology

[0002] Mixture of Experts (MoE) models boast a powerful model architecture and are widely used in fields such as natural language processing and computer vision, providing services like machine translation, text generation, and question answering. The core characteristic of MoE models is the inclusion of multiple distinct expert networks, each dedicated to handling a specific type of task. When a model receives a complex task, it can decompose it into multiple sub-tasks of different types, assigning each sub-task to the corresponding expert network for processing. This significantly improves inference speed and accuracy.

[0003] However, as the scale of hybrid expert models continues to expand, the number of expert networks and parameters in the models also gradually increases. During the model operation, excessive computing resources are often required, which greatly limits the application of the models in resource-constrained environments (such as edge devices) and vertical fields. Summary of the Invention

[0004] This application provides a model processing method, a model processing apparatus, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem that hybrid expert models in related technologies consume excessive computing resources.

[0005] This application provides a model processing method, including:

[0006] Obtain the network weight matrix of the hybrid expert model, wherein the hybrid expert model includes multiple expert networks, the columns of the network weight matrix correspond one-to-one with the expert networks, and each column is used to characterize the activation mode of the corresponding expert network.

[0007] Perform singular value decomposition on the network weight matrix to obtain the right singular vector matrix;

[0008] In the right singular vector matrix, the columns with singular values ​​less than a threshold are used as the null basis matrix, and a sub-network weight matrix is ​​constructed based on the null basis matrix and each column of the network weight matrix.

[0009] Singular value decomposition is performed on the weight matrix of the sub-network to obtain the maximum singular value and the minimum singular value;

[0010] Based on the maximum singular value and the minimum singular value, the invalid feature sensitivity of each of the expert networks is determined, and based on the invalid feature sensitivity, the network parameters of the hybrid expert model are pruned.

[0011] This application also provides a model processing apparatus, including:

[0012] The weight matrix acquisition module is used to acquire the network weight matrix of the hybrid expert model, which includes multiple expert networks. The columns of the network weight matrix correspond one-to-one with the expert networks, and each column is used to characterize the activation mode of the corresponding expert network.

[0013] The first singular value decomposition module is used to perform singular value decomposition on the network weight matrix to obtain the right singular vector matrix.

[0014] The weight matrix reconstruction module is used to take the columns with singular values ​​less than a threshold as null basis matrices in the right singular vector matrix, and construct a sub-network weight matrix based on the null basis matrix and each column of the network weight matrix.

[0015] The second singular value decomposition module is used to perform singular value decomposition on the sub-network weight matrix to obtain the maximum singular value and the minimum singular value.

[0016] The parameter pruning module is used to determine the invalid feature sensitivity of each of the expert networks based on the maximum singular value and the minimum singular value, and to prune the network parameters of the hybrid expert model based on the invalid feature sensitivity.

[0017] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of the above-described model processing method when executing the computer program.

[0018] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described model processing method.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described model processing method.

[0020] In some embodiments of this application, singular value decomposition is performed on the network weight matrix of the hybrid expert model, and columns with singular values ​​less than a threshold are selected from the resulting right singular vector matrix as the null basis matrix. Thus, the null basis matrix can characterize activation mode directions that contribute little to the activation of the expert network, or activation mode directions that contribute nothing. Furthermore, a sub-network weight matrix is ​​constructed based on the columns of the null basis matrix and the network weight matrix. In this way, the degree of linear correlation between each column of the network weight matrix and the null basis matrix can be determined based on the maximum and minimum singular values ​​of the sub-network weight matrix. Furthermore, the ineffective feature sensitivity of each expert network can be determined based on the degree of linear correlation, and the network parameters of the hybrid expert model are pruned based on the ineffective feature sensitivity. Compared to some techniques that prune expert networks based on activation counts or weight norms, this application prunes the model's network parameters based on the ineffective feature sensitivity of each expert network. This significantly reduces the false deletion rate of the expert network while reducing the number of model parameters, thereby ensuring that model performance is not affected. In summary, the method of this application can solve the problem that hybrid expert models in related technologies consume too much computing resources. Attached Figure Description

[0021] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A schematic flowchart illustrating the model processing method provided for some embodiments of this application;

[0023] Figure 2 A schematic diagram of a model processing apparatus provided for some embodiments of this application;

[0024] Figure 3 A schematic diagram of the modules of an electronic device provided for some embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0026] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] Hybrid expert models can be used for image processing or text processing. The following example illustrates the inference principle of a hybrid expert model. For instance, in a hybrid expert model for image recognition, expert networks N1 to N3 can be set up. Expert network N1 is used to identify animals in the image, expert network N2 is used to identify buildings in the image, and expert network N3 is used to identify people in the image. After receiving an image recognition task, the model can generate three sub-tasks A1 to A3 based on the image to be recognized and perform the following operations: sub-task A1 is assigned to expert network N1 to identify whether animals exist in the image; sub-task A2 is assigned to expert network N2 to identify whether buildings exist in the image; and sub-task A3 is assigned to expert network N3 to identify whether people exist in the image. In this way, on the one hand, the inference complexity of a single expert network can be reduced, thereby greatly improving the overall inference accuracy of the model; on the other hand, multiple expert networks can infer in parallel, improving the inference speed of the model.

[0029] However, with the development of technology, the scale of hybrid expert models continues to expand, and the number of expert networks and parameters in the models are also gradually increasing. This leads to increasingly higher requirements for computing resources, which in turn greatly limits the application of models in resource-constrained environments (such as edge devices) and vertical fields.

[0030] To address the aforementioned issues, some techniques have proposed model parameter pruning schemes. Specifically, these techniques prune expert networks with fewer activations from the trained hybrid expert model based on the number of times each expert network is activated. This reduces the number of model parameters and lowers the computational resource requirements. However, a low activation count for an expert network does not necessarily mean it is useless; it may simply mean that the task it performs is rare. Pruning such an expert network would prevent the hybrid expert model from performing the target task, thus reducing model performance.

[0031] In other techniques, expert networks with lower norms are pruned based on the weight norms of each expert network. However, a lower norm does not necessarily mean that the expert network is redundant; therefore, arbitrarily pruning these expert networks will also reduce model performance.

[0032] Therefore, this application provides a model processing method that can reduce the number of expert networks and parameters in a hybrid expert model while maintaining model performance, thereby reducing the model's computational resource consumption. The model processing method can be applied to electronic devices running hybrid expert models. These electronic devices may include, but are not limited to, tablets, laptops, desktop computers, and servers. (See also...) Figure 1 This is a flowchart illustrating the model processing method provided in some embodiments of this application. Figure 1 In this model processing method, the following steps are included:

[0033] Step S101: Obtain the network weight matrix of the hybrid expert model. The hybrid expert model includes multiple expert networks. The columns of the network weight matrix correspond one-to-one with the expert networks, and each column is used to represent the activation mode of the corresponding expert network.

[0034] Specifically, expression (1) gives an example of the network weight matrix W.

[0035] W= (1)

[0036] As shown in expression (1), the number of rows in the network weight matrix equals the input feature dimension of the hybrid expert model, and the number of columns equals the number of expert networks in the hybrid expert model. Simultaneously, each row of the network weight matrix corresponds one-to-one with the input feature dimension of the hybrid expert model, with each row representing the activation contribution weight of the corresponding input feature dimension to each expert network. Furthermore, each column of the network weight matrix corresponds to one of the expert networks in the hybrid expert model, and each column represents the activation mode of the corresponding expert network. For ease of understanding, examples are provided below.

[0037] For example, in expression (1), the first column can correspond to the first expert network, the second column can correspond to the second expert network, and so on. The input feature of the hybrid expert model is a 4-dimensional vector. For the vector in the first row In other words, it represents the input features. The activation contribution weight for the first expert network is 2, the activation contribution weight for the second expert network is 1, and the activation contribution weight for the third expert network is 0. For the vector in the second row... In other words, it represents the input features. The activation contribution weight for the first expert network is 1, the activation contribution weight for the second expert network is 3, the activation contribution weight for the third expert network is 1, and so on.

[0038] Similarly, for the vector in the first column... In other words, it represents the input features. The activation contribution weight of the first expert network is 2, and the input features are... The activation contribution weight of the first expert network is 1, and the input features are... The activation contribution weight of the first expert network is 0, and the input features are... The activation contribution weight for the first expert network is 1. For the vector in the second column... In other words, it represents the input features. The activation contribution weight of the second expert network is 1, and the input features are... The activation contribution weight of the second expert network is 3, and the input features are... The activation contribution weight of the second expert network is 1, and the input features are... The activation contribution weight for the second expert network is 0. And so on.

[0039] Step S102: Perform singular value decomposition on the network weight matrix to obtain the right singular vector matrix.

[0040] Specifically, after performing singular value decomposition on the network weight matrix, we obtain the left singular vector matrix U, the singular value matrix Σ, and the right singular vector matrix V. The left singular vector matrix U represents the orthogonal basis of the input feature vectors, and the right singular vector matrix V represents the activation mode of the expert network. The singular values ​​in the singular value matrix Σ correspond one-to-one with the columns of the left singular vector matrix U, i.e., the singular values... The i-th column corresponds to the left singular vector matrix U. Similarly, the singular values ​​in the singular value matrix Σ also correspond one-to-one with the columns of the right singular vector matrix V, i.e., the singular values... This corresponds to the i-th column of the right singular vector matrix V. The number of non-zero singular values ​​in the singular value matrix Σ is equal to the rank of the network weight matrix. Here, rank refers to the maximum number of linearly independent rows or columns in the matrix. For example, suppose the network weight matrix has 5 rows, where a linear transformation of the third row vector yields the fourth row vector. Then the rank of the network weight matrix is ​​4, indicating that there are 4 linearly independent rows.

[0041] The left singular vector matrix U is used to project the original input feature vectors of the hybrid expert model onto a low-dimensional orthogonal space to obtain features without redundancy. The singular value matrix Σ is used to scale the orthogonally projected features to control the activation contribution weights of different orthogonal bases to the expert networks. The right singular vector matrix V is used to map the scaled features to the activation intensities of each expert network. For ease of understanding, the network weight matrix in expression (1) is used as an example.

[0042] Specifically, after performing singular value decomposition on the network weight matrix in expression (1), the left singular vector matrix U, the singular value matrix Σ, and the right singular vector matrix V are obtained as shown in expressions (2) to (4).

[0043] (2)

[0044] (3)

[0045] (4)

[0046] Specifically, in the singular value matrix Σ, the singular values ​​are located on the diagonal of the matrix. Taking the above expression (3) as an example, the first singular value in the upper left corner of the matrix... =4.0000, the second singular value =2.4495, the third singular value = It should be noted that although there are 4 rows in the network weight matrix shown in expression (1), there are only 3 non-zero singular values ​​in the singular value matrix Σ because two of the rows are linearly dependent. At the same time, in order to ensure that the dimensions of the left singular vector matrix U and the singular value matrix Σ are aligned, a row of 0 is added at the end of the singular value matrix Σ for subsequent calculation.

[0047] Based on the above description, assuming the original input feature vector of the hybrid expert model is... Multiplying X by U yields the projected features. Then the projected feature X1 and After multiplication, the scaled features can be obtained. Finally, multiplying X2 by the transpose of V yields the activation strength of each expert network. .

[0048] Step S103: In the right singular vector matrix, the columns with singular values ​​less than the threshold are used as the null basis matrix, and the sub-network weight matrix is ​​constructed based on the columns of the null basis matrix and the network weight matrix.

[0049] Specifically, based on the "singular values" described in step S101 above... "Corresponding to the i-th column of the right singular vector matrix V", we can search in the right singular vector matrix of step S103 for whether there is a column with singular values ​​less than the threshold. If so, we will form a null basis matrix with one or more of the corresponding columns. For ease of understanding, we will take the singular value matrix Σ shown in expression (5) as an example.

[0050] (5)

[0051] Assuming expression (5) , , , , And if the threshold is 0.6, then the right singular vector matrix V will be... The three corresponding columns form the null basis matrix.

[0052] Based on the calculation process in step S102 above, it can be understood that columns in the right singular vector matrix V with singular values ​​not less than a threshold can be considered as useful expert activation mode directions, and columns in the right singular vector matrix V with singular values ​​less than a threshold can be considered as useless expert activation mode directions. Correspondingly, the input feature vector X can be decomposed into effective features and invalid features, where effective features are mapped to columns in the right singular vector matrix V with singular values ​​not less than a threshold, and invalid features are mapped to columns in the right singular vector matrix V with singular values ​​less than a threshold.

[0053] Therefore, the null basis matrix composed of columns with singular values ​​less than the threshold in the right singular vector matrix can be regarded as the null space of expert activation modes. This null space can be regarded as the activation mode direction that contributes little to the activation of the expert network, or the activation mode direction that does not contribute to the activation of the expert network.

[0054] Furthermore, after obtaining the null basis matrix, any column in the network weight matrix where the singular value is not less than a threshold can be taken as the target column. This target column is then merged with the null basis matrix to obtain the sub-network weight matrix corresponding to the target expert network. The target column corresponds to the target expert network in the hybrid expert model.

[0055] Taking the null basis matrix obtained based on the above expression (5) as an example. For instance, a sub-network weight matrix W1 can be constructed based on the first column of the network weight matrix and the null basis matrix. Since the first column of the network weight matrix corresponds to the first expert network, the sub-network weight matrix W1 can also be regarded as the sub-network weight matrix corresponding to the first expert network. For another example, a sub-network weight matrix W2 can be constructed based on the second column of the network weight matrix and the null basis matrix. Since the second column of the network weight matrix corresponds to the second expert network, the sub-network weight matrix W2 can also be regarded as the sub-network weight matrix corresponding to the second expert network. In this way, a sub-network weight matrix can be constructed for each expert network.

[0056] Step S104: Perform singular value decomposition on the sub-network weight matrix to obtain the maximum singular value and the minimum singular value.

[0057] Specifically, singular value decomposition can be performed on the weight matrix of the sub-networks corresponding to the target expert network to obtain the corresponding maximum and minimum singular values. The target expert network can be any expert network. That is, in step S104, the maximum and minimum singular values ​​corresponding to each expert network can be obtained.

[0058] Step S105: Based on the maximum and minimum singular values, determine the ineffective feature sensitivity of each expert network, and based on the ineffective feature sensitivity, prune the network parameters of the hybrid expert model.

[0059] In this embodiment, for any expert network, the sensitivity to invalid features can be determined based on the maximum and minimum singular values ​​corresponding to that expert network. The sensitivity to invalid features refers to the degree to which an expert network depends on invalid features when activated. Specifically, if an expert network primarily relies on invalid features when activated, its sensitivity to invalid features is high; conversely, if an expert network primarily relies on valid features when activated, its sensitivity to invalid features is low.

[0060] Specifically, in step S104, since a corresponding sub-network weight matrix is ​​constructed for each expert network, and the sub-network weight matrix consists of the network weight matrix columns corresponding to the expert network and the null basis matrix, the ineffective feature sensitivity of each expert network can be determined by performing singular value decomposition on each sub-network weight matrix and based on the maximum and minimum singular values ​​obtained from the decomposition. Simply put, after performing singular value decomposition on the sub-network weight matrix corresponding to any expert network, if there are small singular values, it indicates that the network weight matrix columns corresponding to that expert network have a nearly linear relationship with the null basis matrix (e.g., the minimum singular value is 0.01), or that the network weight matrix columns corresponding to that expert network have a linear relationship with the null basis matrix (e.g., the minimum singular value is 0).

[0061] When the network weight matrix of an expert network is nearly linearly related to or has a linear relationship with the null basis matrix, it means that the activation mode direction of the expert network is highly coincident with or the same as the activation mode direction of the null basis matrix. Furthermore, since the activation mode direction of the null basis matrix mainly depends on invalid features, the activation mode direction of the expert network also mainly depends on invalid features, thus the expert network has high sensitivity to invalid features.

[0062] Conversely, when the network weight matrix of an expert network exhibits a non-linear relationship with the null basis matrix, it means that the activation mode direction of the expert network is different from that of the null basis matrix. Furthermore, since the activation mode direction of the null basis matrix primarily depends on invalid features, it can be inferred that the activation mode direction of the expert network mainly relies on invalid features, thus indicating low sensitivity to invalid features in the expert network.

[0063] Furthermore, after obtaining the ineffective feature sensitivity of each expert network, the expert networks can be sorted according to the order of their ineffective feature sensitivity to obtain a sorted sequence. On the one hand, since expert networks with high ineffective feature sensitivity mainly rely on ineffective feature activation, meaning that their activation does not affect the output of the hybrid expert model, these expert networks can be considered redundant. On the other hand, since expert networks with low ineffective feature sensitivity mainly rely on effective feature activation, meaning that their activation will affect the output of the hybrid expert model, these expert networks can be considered effective. When pruning model parameters, redundant expert networks can be pruned, while effective expert networks are retained. Compared to some techniques that prune expert networks based on activation count or weight norm, the method in this application does not mistakenly delete expert networks, meaning that it can reduce the number of model parameters while ensuring that the performance of the hybrid expert model is not affected, thereby solving the problem of excessive computational resources required by hybrid expert models in related techniques.

[0064] In some embodiments of this application, singular value decomposition is performed on the network weight matrix of the hybrid expert model, and columns with singular values ​​less than a threshold are selected from the resulting right singular vector matrix as the null basis matrix. Thus, the null basis matrix can characterize activation mode directions that contribute little to the activation of the expert network, or activation mode directions that contribute nothing. Furthermore, a sub-network weight matrix is ​​constructed based on the columns of the null basis matrix and the network weight matrix. In this way, the degree of linear correlation between each column of the network weight matrix and the null basis matrix can be determined based on the maximum and minimum singular values ​​of the sub-network weight matrix. Furthermore, the ineffective feature sensitivity of each expert network can be determined based on the degree of linear correlation, and the network parameters of the hybrid expert model are pruned based on the ineffective feature sensitivity. Compared to some techniques that prune expert networks based on activation counts or weight norms, this application prunes the model's network parameters based on the ineffective feature sensitivity of each expert network. This significantly reduces the false deletion rate of the expert network while reducing the number of model parameters, thereby ensuring that model performance is not affected. In summary, the method of this application can solve the problem that hybrid expert models in related technologies consume too much computing resources.

[0065] In some embodiments, step S105, which determines the ineffective feature sensitivity of the target expert network based on the maximum and minimum singular values, includes:

[0066] A mapping relationship between the maximum singular value, the minimum singular value, and the invalid feature sensitivity is constructed, and based on the mapping relationship, the invalid feature sensitivity of the target expert network is obtained. In the mapping relationship, the maximum singular value is directly proportional to the invalid feature sensitivity, and the minimum singular value is inversely proportional to the invalid feature sensitivity.

[0067] Specifically, the target expert network is any expert network of the hybrid expert model. The mapping relationship between the maximum singular value, the minimum singular value and the sensitivity to invalid features can be shown in expression (6).

[0068] (6)

[0069] in, Indicates sensitivity to invalid features. Represents the maximum singular value. This represents the smallest singular value. The larger the value of , the higher the sensitivity to invalid features.

[0070] For any expert network, singular value decomposition can be performed on the weight matrix of the corresponding subnetwork to obtain the corresponding maximum and minimum singular values. These maximum and minimum singular values ​​are then substituted into the above expression (6) to obtain the ineffective feature sensitivity of the expert network. It is understandable that since the weight matrices of the subnetworks corresponding to different expert networks are different, the ineffective feature sensitivity of different expert networks can be different. Therefore, expert networks can be pruned according to the order of their ineffective feature sensitivity.

[0071] It should be noted that expression (6) is merely an example of a mapping relationship and does not constitute a limitation on this application. In practical applications, any mapping relationship that satisfies the condition that the maximum singular value is directly proportional to the sensitivity to invalid features and the minimum singular value is inversely proportional to the sensitivity to invalid features should fall within the protection scope of this application.

[0072] In the above embodiments, a reasonable mapping relationship between the maximum singular value, the minimum singular value, and the invalid feature sensitivity is constructed, and based on this mapping relationship, the maximum singular value and the minimum singular value are converted into invalid feature sensitivity. In this way, it can be ensured that the invalid feature sensitivity of multiple expert networks is determined based on the same logic, avoiding the accidental deletion of expert networks, and thus ensuring the performance of the hybrid expert model.

[0073] In some embodiments, the hybrid expert model may include multiple inference layers, each comprising multiple expert networks. Specifically, the number and type of expert networks in different inference layers may not be entirely the same. For example, the inference layer for low-dimensional feature extraction may include more expert networks, while the inference layer for high-dimensional feature extraction may include fewer expert networks. Furthermore, in the inference layer for low-dimensional feature extraction, each expert network may primarily extract specific types of low-dimensional features, and in the inference layer for high-dimensional feature extraction, each expert network may primarily extract specific types of high-dimensional features.

[0074] Since the number and type of expert networks in different inference layers are not entirely the same, evaluating and pruning expert networks in different inference layers according to the same invalid feature sensitivity criterion may lead to inaccuracies. For example, suppose inference layer C1 includes expert networks N1 to N3, where the invalid feature sensitivities of expert networks N1 to N3 are 4, 5, and 8 respectively, and inference layer C1 is an inference layer that cannot be pruned, meaning that at least one expert network needs to be retained in inference layer C1. However, when sorting the expert networks of all inference layers according to invalid feature sensitivity, the invalid feature sensitivities of expert networks N1 to N3 may all be in a high range and need to be pruned. This would cause inaccurate pruning. In view of this, the method of this application may also include:

[0075] Obtain the layer network weight matrix of the target inference layer. The columns of the layer network weight matrix correspond one-to-one with the expert network of the target inference layer. Each column is used to represent the activation mode of the corresponding expert network.

[0076] Based on the layer network weight matrix, the ineffective feature sensitivity of each expert network in the target inference layer is determined, and the network parameters of the target inference layer are pruned based on the ineffective feature sensitivity.

[0077] Specifically, the target inference layer is any inference layer of the hybrid expert model.

[0078] In this embodiment, it is equivalent to performing steps S101 to S105 respectively based on the layer network weight matrix of each inference layer. For example, assuming that the hybrid expert model includes inference layers C1 to C3, and inference layer C1 includes expert networks N1 to N3, inference layer C2 includes expert networks N4 to N8, and inference layer C3 includes expert networks N9 to N15, and inference layer C1 has a layer network weight matrix T1, inference layer C2 has a layer network weight matrix T2, and inference layer C3 has a layer network weight matrix T3, then the expert networks of each inference layer can be pruned according to the following steps:

[0079] Perform the following steps 11) to 14) to prune the expert network of inference layer C1.

[0080] 11) Perform singular value decomposition on the weight matrix T1 of the layer network to obtain the right singular vector matrix V1;

[0081] 12) In the right singular vector matrix V1, the columns with singular values ​​less than the threshold are taken as the null basis matrix T1, and the sub-network weight matrix is ​​constructed based on the columns of the null basis matrix T1 and the layer network weight matrix T1.

[0082] 13) Perform singular value decomposition on the subnetwork weight matrix from step 12) to obtain the maximum and minimum singular values;

[0083] 14) Based on the maximum and minimum singular values ​​in step 13), obtain the invalid feature sensitivities M1 to M3 of expert networks N1 to N3, and prune the expert network of inference layer C1 based on the invalid feature sensitivities M1 to M3.

[0084] Perform the following steps 21) to 24) to prune the expert network of inference layer C2.

[0085] 21) Perform singular value decomposition on the weight matrix T2 of the layer network to obtain the right singular vector matrix V2;

[0086] 22) In the right singular vector matrix V2, the columns with singular values ​​less than the threshold are taken as the null basis matrix T2, and the sub-network weight matrix is ​​constructed based on the columns of the null basis matrix T2 and the layer network weight matrix T2.

[0087] 23) Perform singular value decomposition on the subnetwork weight matrix from step 22) to obtain the maximum and minimum singular values;

[0088] 24) Based on the maximum and minimum singular values ​​in step 23), obtain the invalid feature sensitivity M4~M8 of expert networks N4~N8, and prune the expert network of inference layer C2 based on the invalid feature sensitivity M4~M8.

[0089] Following steps 1) or 2), the expert network of inference layer C3 can be further pruned.

[0090] In the above embodiments, the expert network of the target inference layer is pruned based on the layer network weight matrix of the target inference layer. This avoids the problem of erroneous deletion of expert networks caused by differences in expert networks across different inference layers, thus ensuring that the performance of the hybrid expert model is not affected by network parameter pruning.

[0091] Similar to the embodiments described above, the thresholds for at least some inference layers can be different. Thus, different thresholds can be set for different inference layers based on their actual conditions. This allows for the selection of columns used to construct the null basis matrix from the right singular vector matrix corresponding to each inference layer according to different thresholds, enabling the pruning of different numbers of expert networks at different inference layers. Specifically, more expert networks can be pruned in inference layers with smaller thresholds, and fewer expert networks can be pruned in inference layers with larger thresholds. Based on the above description, the determination of the ineffective feature sensitivity of each expert network in the target inference layer based on the layer network weight matrix can include:

[0092] Obtain the target threshold corresponding to the target inference layer;

[0093] Singular value decomposition is performed on the weight matrix of the layer network to obtain the right singular vector matrix;

[0094] In the obtained right singular vector matrix, the columns with singular values ​​less than the target threshold are used as the layer null space basis matrix. Based on the columns of the layer null space basis matrix and the layer network weight matrix, the layer sub-network weight matrix is ​​constructed. Based on the layer sub-network weight matrix, the invalid feature sensitivity of each expert network in the target inference layer is determined.

[0095] Similar to the above embodiments, the target inference layer can be any inference layer of the hybrid expert model. For ease of understanding, the following example illustrates this. For instance, suppose the hybrid expert model includes inference layers C1 to C3, where inference layer C1 includes expert networks N1 to N3, inference layer C2 includes expert networks N4 to N8, and inference layer C3 includes expert networks N9 to N15. Inference layer C1 has a layer network weight matrix T1 and a threshold P1, inference layer C2 has a layer network weight matrix T2 and a threshold P2, and inference layer C3 has a layer network weight matrix T3 and a threshold P3. Since thresholds P1, P2, and P3 are not the same, the expert networks of each inference layer can be pruned based on the following steps:

[0096] Perform the following steps 31) to 34) to prune the expert network of inference layer C1.

[0097] 31) Perform singular value decomposition on the weight matrix T1 of the layer network to obtain the right singular vector matrix V1;

[0098] 32) In the right singular vector matrix V1, the columns with singular values ​​less than the threshold P1 are taken as the null basis matrix T1, and the sub-network weight matrix is ​​constructed based on the columns of the null basis matrix T1 and the layer network weight matrix T1.

[0099] 33) Perform singular value decomposition on the subnetwork weight matrix from step 32) to obtain the maximum and minimum singular values;

[0100] 34) Based on the maximum and minimum singular values ​​in step 33), obtain the invalid feature sensitivities M1 to M3 of expert networks N1 to N3, and prune the expert network of inference layer C1 based on the invalid feature sensitivities M1 to M3.

[0101] Perform the following steps 41) to 44) to prune the expert network of inference layer C2.

[0102] 41) Perform singular value decomposition on the weight matrix T2 of the layer network to obtain the right singular vector matrix V2;

[0103] 42) In the right singular vector matrix V2, the columns with singular values ​​less than the threshold P2 are taken as the null basis matrix T2, and the sub-network weight matrix is ​​constructed based on the columns of the null basis matrix T2 and the layer network weight matrix T2.

[0104] 43) Perform singular value decomposition on the subnetwork weight matrix from step 42) to obtain the maximum and minimum singular values;

[0105] 44) Based on the maximum and minimum singular values ​​in step 43), the invalid feature sensitivities M4 to M8 of the expert networks N4 to N8 are obtained, and the expert network of the inference layer C2 is pruned based on the invalid feature sensitivities M4 to M8.

[0106] In the above embodiments, different thresholds are set for different inference layers. In this way, the number of expert networks pruned for each inference layer can be controlled by adjusting the thresholds according to the importance of each inference layer or the actual situation, thereby further improving the pruning accuracy of the expert networks.

[0107] For ease of understanding, several specific examples of threshold settings are given below.

[0108] In some embodiments, obtaining the target threshold corresponding to the target inference layer includes:

[0109] The target threshold is determined based on the feature dimensions extracted by the target inference layer during the inference process. The lower the feature dimensions extracted by the target inference layer, the larger the target threshold.

[0110] In these embodiments, the inference layer used to extract low-dimensional features can have a larger threshold. This allows for a reduction in the number of expert network pruning steps within the corresponding inference layer, thereby extracting more low-dimensional features. Conversely, in the inference layer used to extract high-dimensional features, the number of expert networks has little impact on model performance. Therefore, the corresponding inference layer can have a smaller threshold, allowing for an increase in the number of expert network pruning steps within the corresponding inference layer, thus reducing the number of model parameters.

[0111] In some embodiments, obtaining the target threshold corresponding to the target inference layer includes:

[0112] Obtain the divergence between the input feature distribution and the output feature distribution of the target inference layer. The divergence is used to characterize the degree of difference between the input feature distribution and the output feature distribution.

[0113] The target threshold is determined based on the divergence, where the smaller the divergence, the larger the target threshold.

[0114] Specifically, for any inference layer, if the divergence between the input and output feature distributions is small, it indicates that the features have not changed significantly after passing through the inference layer. In this case, the importance of each expert network in the inference layer cannot be accurately determined. Therefore, the threshold for this inference layer can be set to a larger threshold to reduce the number of expert networks pruned, thereby avoiding the accidental deletion of expert networks.

[0115] In some embodiments, obtaining the target threshold corresponding to the target inference layer includes:

[0116] Obtain the first number of times the expert network of the target inference layer is activated;

[0117] Take any other inference layer besides the target inference layer as the reference inference layer, and obtain the second number of times the expert network of the reference inference layer is activated;

[0118] If the second number is greater than the first number, then the target threshold is less than the threshold of the reference inference layer; if the second number is less than the first number, then the target threshold of the target inference layer is greater than the threshold of the reference inference layer.

[0119] Specifically, for any inference layer, if the inference layer is activated a lot, it can be preliminarily judged that the inference layer is a relatively important inference layer. In this case, the threshold of the inference layer can be set to a larger threshold, so as to reduce the number of expert network prunings for the inference layer and thus avoid the situation of expert network being deleted by mistake.

[0120] Furthermore, when determining whether the first activation count of an expert network in an inference layer is relatively high, the second activation count of the expert network in any other inference layer can be obtained, and the first and second counts can be compared. If the first count is greater than the second count, it indicates that the expert network in that inference layer has been activated more times than other inference layers; if the first count is less than the second count, it indicates that the expert network in that inference layer has been activated less times than other inference layers. By comparing with other inference layers, it is possible to objectively and accurately assess whether the first activation count of an inference layer's expert network is relatively high, thus providing a more accurate reference for threshold setting.

[0121] In some embodiments, the average number of times the expert network of all inference layers is activated can be calculated. To determine whether the first activation count of an expert network in an inference layer is relatively high, the first activation count can be compared to the average count. If the first activation count is greater than the second, it indicates that the expert network of that inference layer has been activated a relatively high number of times; if the first activation count is less than the second, it indicates that the expert network of that inference layer has been activated a relatively low number of times. This averaging method can also provide a more accurate reference for threshold setting.

[0122] In some embodiments, obtaining the target threshold corresponding to the target inference layer includes:

[0123] Obtain the singular value distribution of the layer network weight matrix corresponding to the target inference layer;

[0124] In a singular value distribution, determine the difference between any two adjacent singular values;

[0125] The target threshold is determined based on the volatility of the difference, where the greater the volatility of the difference, the smaller the target threshold.

[0126] Specifically, let's take the inference layer C1 as an example. Assume the layer network weight matrix of inference layer C1 is T1. To determine the difference between any two adjacent singular values ​​in the layer network weight matrix T1, we can subtract the two adjacent singular values. For example, ... and Subtraction yields the result. and The difference D1. and Subtraction yields the result. and The difference is D2. In this way, multiple differences can be obtained.

[0127] If the differences among the obtained values ​​fluctuate significantly, it indicates that there are abrupt changes in the singular values ​​of the layer network weight matrix T1. For example, and When the difference D2 is large, it indicates that in Afterward, the singular values ​​will decrease significantly. When the difference fluctuates greatly, it indicates that in the inference layer C1, only a small number of expert network activations depend on valid features, while the activations of most expert networks depend on invalid features. In this case, a smaller threshold can be set for the inference layer C1 to increase the number of expert network prunings in the inference layer C1, thereby reducing the number of model parameters.

[0128] Conversely, among the obtained differences, if the fluctuation of the difference is relatively small, it indicates that the singular values ​​of the layer network weight matrix T1 change relatively smoothly. When the singular values ​​change relatively smoothly, it means that in the inference layer C1, most of the expert network activations rely on effective features. In this case, a larger threshold can be set for the inference layer C1 to reduce the number of expert network prunings in the inference layer C1, thereby reducing the problem of accidental deletion of expert networks.

[0129] In the above embodiments, since the fluctuation of the difference between adjacent singular values ​​can reflect the number of expert networks that rely on effective features in each inference layer, determining the threshold of each inference layer based on the fluctuation of the difference between adjacent singular values ​​can greatly reduce the probability of expert networks being mistakenly deleted, and at the same time, can also effectively reduce the number of model parameters.

[0130] In some embodiments, determining the target threshold based on the volatility of the difference includes:

[0131] If there is a difference greater than the difference threshold, then among the obtained differences, find the maximum difference and the two adjacent singular values ​​of the maximum difference;

[0132] Among the two adjacent singular values ​​found, the value that is less than or equal to the smallest singular value is used as the target threshold.

[0133] Specifically, the difference threshold is used to assess the volatility of the difference values. For any inference layer, if any of the singular value differences of that inference layer exceeds the difference threshold, it indicates that the singular value differences of that inference layer have significant volatility. Conversely, if none of the singular value differences of that inference layer exceed the difference threshold, it indicates that the singular value differences of that inference layer are relatively flat.

[0134] Furthermore, for any inference layer, if the singular value difference of the inference layer has large fluctuations, the two adjacent singular values ​​corresponding to the largest difference can be found, and these two adjacent singular values ​​can be used as the singular value abrupt change locations. For example, ... and Subtracting them gives us the difference D1. and Subtracting them gives us the difference D2. and Subtracting them gives the difference D3. If the difference D3 is the maximum difference, then... and The intervals between these points represent singular value mutation locations.

[0135] After finding the location of a singular value abrupt change, the smaller singular value at that location can be taken as the smallest singular value. For example, using the above... and For example, we can As the minimum singular value. When determining the threshold, it can be... As a threshold, or less than The value is used as the threshold.

[0136] In the above embodiments, among the two adjacent singular values ​​corresponding to the maximum difference, the value that is less than or equal to the minimum singular value is used as the threshold, thus obtaining a more accurate threshold.

[0137] In some embodiments, the above-mentioned pruning of network parameters of the target inference layer based on ineffective feature sensitivity may include:

[0138] Obtain the number of expert network prunes for the target inference layer;

[0139] Using the number of expert network prunings as a constraint, the expert network of the target inference layer is pruned sequentially according to the order of sensitivity to invalid features from high to low.

[0140] Specifically, as mentioned above, the target inference layer can be any inference layer of the hybrid expert model. Each inference layer can have its own corresponding number of expert network prunings, thus avoiding accidental or missed deletions of expert networks.

[0141] It should be noted that in the above embodiments, the number of expert networks pruned in each inference layer was indirectly controlled by setting different thresholds for different inference layers. However, in this embodiment, the number of expert networks pruned in each inference layer is directly specified. For example, after setting the number of expert networks pruned in inference layer C1 to 3, the expert networks of inference layer C1 can be sorted in descending order of invalid feature sensitivity, and the top 3 expert networks with the highest invalid feature sensitivity can be pruned.

[0142] In the above embodiments, after setting the number of expert network prunings for each inference layer, the number of expert network prunings for each inference layer can be precisely controlled according to actual needs, so that the number of expert network prunings meets the actual requirements.

[0143] The following examples illustrate how to determine the number of expert network prunes.

[0144] In some embodiments, obtaining the number of expert network prunes for the target inference layer includes:

[0145] Obtain the total number of expert networks and the pruning ratio of the target inference layer. The lower the feature dimension extracted by the target inference layer during the inference process, the lower the pruning ratio; or the more times the expert network of the target inference layer is activated, the lower the pruning ratio.

[0146] The number of expert networks to be pruned is determined based on the total number of expert networks and the pruning ratio.

[0147] Specifically, the pruning ratio can vary across different inference layers. The pruning ratio for each inference layer is based on the total number of expert networks in that layer. For example, if inference layer C1 includes 10 expert networks and the pruning ratio is 10%, then the number of expert networks pruned in inference layer C1 is 1. As another example, if inference layer C2 includes 20 expert networks and the pruning ratio is 20%, then the number of expert networks pruned in inference layer C2 is 4.

[0148] The logic for determining the pruning ratio of each inference layer is similar to the threshold determination logic described above, and will not be elaborated here.

[0149] In some embodiments, obtaining the number of expert network prunes for the target inference layer includes:

[0150] In the expert network of the target inference layer, find the number of expert networks whose sensitivity to invalid features exceeds the sensitivity threshold;

[0151] Obtain the divergence between the input feature distribution and the output feature distribution of the target inference layer. The divergence is used to characterize the degree of difference between the input feature distribution and the output feature distribution.

[0152] The number of expert networks to be pruned is determined based on the divergence and the number of expert networks exceeding the sensitivity threshold. The smaller the divergence, the fewer the number of networks to be pruned, and the more expert networks that exceed the sensitivity threshold, the fewer the number of networks to be pruned.

[0153] Specifically, the logic for determining the number of expert network pruning operations in this embodiment is similar to the threshold determination logic described above, and will not be repeated here.

[0154] In some embodiments, the method of this application further includes:

[0155] If the invalid feature sensitivity of each expert network in the target inference layer is lower than the sensitivity threshold, then the expert networks of the target inference layer will not be pruned.

[0156] Specifically, for any inference layer, if the sensitivity of invalid features of each expert network in that inference layer is lower than the sensitivity threshold, it means that the activation of all expert networks in that inference layer depends on valid features. In this case, all expert networks in that inference layer can be retained to prevent the problem of accidental deletion of expert networks and avoid affecting model performance.

[0157] In some embodiments, after trimming the network parameters of the hybrid expert model, the method of this application further includes:

[0158] In the network weight matrix, the column corresponding to the pruned expert network is deleted to obtain the new network weight matrix;

[0159] Based on the target domain to which the hybrid expert model needs to be applied, obtain domain sample data;

[0160] Based on the new network weight matrix and domain sample data, the hybrid expert model is fine-tuned to adjust the new network weight matrix and / or the classification head of the hybrid expert model.

[0161] Thus, on the one hand, when the pruned expert network has functions that need to be retained, fine-tuning the training can allow the undressed expert network to adjust its learning mechanism and complete some of the functions of the pruned expert network, ensuring model performance. On the other hand, it can adapt the hybrid expert model to the target domain, improving the inference accuracy of the hybrid expert model in the target domain. Furthermore, updating the network weight matrix may disrupt the linear mapping relationship between the original feature input and expert assignments. Fine-tuning the training can reconstruct the linear mapping relationship between the original feature input and expert assignments, thereby ensuring model performance.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0163] See also Figure 2 This is a schematic diagram of a model processing apparatus provided in some embodiments of this application. Figure 2 In this model processing device, the following components are included:

[0164] The weight matrix acquisition module 201 is used to acquire the network weight matrix of the hybrid expert model. The hybrid expert model includes multiple expert networks. The columns of the network weight matrix correspond one-to-one with the expert networks, and each column is used to represent the activation mode of the corresponding expert network.

[0165] The first singular value decomposition module 202 is used to perform singular value decomposition on the network weight matrix to obtain the right singular vector matrix.

[0166] The weight matrix reconstruction module 203 is used to take the columns with singular values ​​less than a threshold as the null basis matrix in the right singular vector matrix, and construct the sub-network weight matrix based on the columns of the null basis matrix and the network weight matrix.

[0167] The second singular value decomposition module 204 is used to perform singular value decomposition on the sub-network weight matrix to obtain the maximum singular value and the minimum singular value.

[0168] The parameter pruning module 205 is used to determine the ineffective feature sensitivity of each expert network based on the maximum and minimum singular values, and to prune the network parameters of the hybrid expert model based on the ineffective feature sensitivity.

[0169] In some embodiments, the hybrid expert model includes multiple inference layers, each inference layer including multiple expert networks; the parameter pruning module 205 is further configured to:

[0170] Obtain the layer network weight matrix of the target inference layer. The columns of the layer network weight matrix correspond one-to-one with the expert network of the target inference layer. Each column is used to represent the activation mode of the corresponding expert network.

[0171] Based on the layer network weight matrix, the ineffective feature sensitivity of each expert network in the target inference layer is determined, and the network parameters of the target inference layer are pruned based on the ineffective feature sensitivity.

[0172] In some embodiments, at least some inference layers have different thresholds; the parameter pruning module 205 is specifically used for:

[0173] Obtain the target threshold corresponding to the target inference layer;

[0174] Singular value decomposition is performed on the weight matrix of the layer network to obtain the right singular vector matrix;

[0175] In the obtained right singular vector matrix, the columns with singular values ​​less than the target threshold are used as the layer null space basis matrix, and the sub-network weight matrix is ​​constructed based on the columns of the layer null space basis matrix and the layer network weight matrix.

[0176] Based on the sub-network weight matrix, the sensitivity of invalid features of each expert network in the target inference layer is determined.

[0177] In some embodiments, the parameter trimming module 205 is specifically used for:

[0178] The target threshold is determined based on the feature dimensions extracted by the target inference layer during the inference process. The lower the feature dimensions extracted by the target inference layer, the larger the target threshold.

[0179] In some embodiments, the parameter trimming module 205 is specifically used for:

[0180] Obtain the divergence between the input feature distribution and the output feature distribution of the target inference layer. The divergence is used to characterize the degree of difference between the input feature distribution and the output feature distribution.

[0181] The target threshold is determined based on the divergence, where the smaller the divergence, the larger the target threshold.

[0182] In some embodiments, the parameter trimming module 205 is specifically used for:

[0183] Obtain the first number of times the expert network of the target inference layer is activated;

[0184] Take any other inference layer besides the target inference layer as the reference inference layer, and obtain the second number of times the expert network of the reference inference layer is activated;

[0185] If the second number is greater than the first number, then the target threshold is less than the threshold of the reference inference layer; if the second number is less than the first number, then the target threshold of the target inference layer is greater than the threshold of the reference inference layer.

[0186] In some embodiments, the parameter trimming module 205 is specifically used for:

[0187] Obtain the singular value distribution of the layer network weight matrix corresponding to the target inference layer;

[0188] In a singular value distribution, determine the difference between any two adjacent singular values;

[0189] The target threshold is determined based on the volatility of the difference, where the greater the volatility of the difference, the smaller the target threshold.

[0190] In some embodiments, the parameter trimming module 205 is specifically used for:

[0191] If there is a difference greater than the difference threshold, then among the obtained differences, find the maximum difference and the two adjacent singular values ​​of the maximum difference;

[0192] Among the two adjacent singular values ​​found, the value that is less than or equal to the smallest singular value is used as the target threshold.

[0193] In some embodiments, the parameter trimming module 205 is specifically used for:

[0194] Obtain the number of expert network prunes for the target inference layer;

[0195] Using the number of expert network prunings as a constraint, the expert network of the target inference layer is pruned sequentially according to the order of sensitivity to invalid features from high to low.

[0196] In some embodiments, the parameter trimming module 205 is specifically used for:

[0197] Obtain the total number of expert networks and the pruning ratio of the target inference layer. The lower the feature dimension extracted by the target inference layer during the inference process, the lower the pruning ratio; or the more times the expert network of the target inference layer is activated, the lower the pruning ratio.

[0198] The number of expert networks to be pruned is determined based on the total number of expert networks and the pruning ratio.

[0199] In some embodiments, the parameter trimming module 205 is specifically used for:

[0200] In the expert network of the target inference layer, find the number of expert networks whose sensitivity to invalid features exceeds the sensitivity threshold;

[0201] Obtain the divergence between the input feature distribution and the output feature distribution of the target inference layer. The divergence is used to characterize the degree of difference between the input feature distribution and the output feature distribution.

[0202] The number of expert networks to be pruned is determined based on the divergence and the number of expert networks exceeding the sensitivity threshold. The smaller the divergence, the fewer the number of networks to be pruned, and the more expert networks that exceed the sensitivity threshold, the fewer the number of networks to be pruned.

[0203] In some embodiments, the parameter trimming module 205 is specifically used for:

[0204] If the invalid feature sensitivity of each expert network in the target inference layer is lower than the sensitivity threshold, then the expert networks of the target inference layer will not be pruned.

[0205] In some embodiments, after pruning the network parameters of the hybrid expert model, the parameter pruning module 205 is specifically used for:

[0206] In the network weight matrix, the column corresponding to the pruned expert network is deleted to obtain the new network weight matrix;

[0207] Based on the target domain to which the hybrid expert model needs to be applied, obtain domain sample data;

[0208] Based on the new network weight matrix and domain sample data, the hybrid expert model is fine-tuned to adjust the new network weight matrix and / or the classification head of the hybrid expert model.

[0209] In some embodiments, the weight matrix reconstruction module 203 is specifically used for:

[0210] In the network weight matrix, any column is taken as the target column, and the target column corresponds to the target expert network of the hybrid expert model;

[0211] The target column and the null basis matrix are merged to obtain the sub-network weight matrix corresponding to the target expert network.

[0212] In some embodiments, the parameter trimming module 205 is specifically used for:

[0213] Singular value decomposition is performed on the weight matrix of the subnetwork corresponding to the target expert network to obtain the corresponding maximum and minimum singular values;

[0214] Based on the obtained maximum and minimum singular values, the sensitivity of invalid features of the target expert network is determined.

[0215] In some embodiments, the parameter trimming module 205 is specifically used for:

[0216] A mapping relationship between the maximum singular value, the minimum singular value, and the invalid feature sensitivity is constructed, and based on the mapping relationship, the invalid feature sensitivity of the target expert network is obtained. In the mapping relationship, the maximum singular value is directly proportional to the invalid feature sensitivity, and the minimum singular value is inversely proportional to the invalid feature sensitivity.

[0217] The model parameter device of this application performs singular value decomposition on the network weight matrix of the hybrid expert model, and selects columns with singular values ​​less than a threshold from the obtained right singular vector matrix as the null basis matrix. Thus, the null basis matrix can represent activation mode directions that contribute little to the activation of the expert network, or activation mode directions that contribute nothing. Furthermore, based on the columns of the null basis matrix and the network weight matrix, a sub-network weight matrix is ​​constructed. Thus, based on the maximum and minimum singular values ​​of the sub-network weight matrix, the degree of linear correlation between each column of the network weight matrix and the null basis matrix can be determined. Furthermore, based on the degree of linear correlation, the ineffective feature sensitivity of each expert network can be determined, and the network parameters of the hybrid expert model can be pruned according to the ineffective feature sensitivity. Compared to some techniques that prune expert networks based on activation count or weight norm, this application prunes the model's network parameters based on the ineffective feature sensitivity of each expert network. This significantly reduces the false deletion rate of expert networks while reducing the number of model parameters, thereby ensuring that model performance is not affected. In summary, the method of this application can solve the problem that hybrid expert models in related technologies consume too much computing resources.

[0218] See also Figure 3 Embodiments of this application also provide an electronic device, including a memory 10 and a processor 20, wherein the memory 10 stores a computer program and the processor 20 is configured to run the computer program to perform the steps in any of the above-described communication method embodiments.

[0219] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described communication method embodiments when it is run.

[0220] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0221] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described communication method embodiments.

[0222] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described communication method embodiments.

[0223] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0224] The foregoing has provided a detailed description of a model processing method, apparatus, device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model processing method characterized by comprising: The method comprises: obtaining a network weight matrix of a mixed expert model, the mixed expert model comprising a plurality of expert networks, columns of the network weight matrix corresponding one-to-one to the expert networks, and each column being used to represent an activation pattern of the corresponding expert network, wherein the mixed expert model is used for image processing and / or text processing; performing singular value decomposition on the network weight matrix to obtain a right singular vector matrix; in the right singular vector matrix, columns with singular values less than a threshold value are taken as a null space basis matrix, and a sub-network weight matrix is constructed based on the null space basis matrix and each column of the network weight matrix; performing singular value decomposition on the sub-network weight matrix to obtain a maximum singular value and a minimum singular value; based on the maximum singular value and the minimum singular value, determining an invalid feature sensitivity of each expert network, and based on the invalid feature sensitivity, pruning network parameters of the mixed expert model to reduce the requirement of the mixed expert model for computing resources.

2. The method of claim 1, wherein, The mixed expert model comprises a plurality of inference layers, and each inference layer comprises a plurality of expert networks; the method further comprises: obtaining a layer network weight matrix of a target inference layer, columns of the layer network weight matrix corresponding one-to-one to expert networks of the target inference layer, and each column being used to represent an activation pattern of the corresponding expert network; based on the layer network weight matrix, determining an invalid feature sensitivity of each expert network of the target inference layer, and based on the invalid feature sensitivity, pruning network parameters of the target inference layer.

3. The method of claim 2, wherein, The threshold values of at least some inference layers are different; the determination of the invalid feature sensitivity of each expert network of the target inference layer based on the layer network weight matrix comprises: obtaining a target threshold value corresponding to the target inference layer; performing singular value decomposition on the layer network weight matrix to obtain a right singular vector matrix; in the obtained right singular vector matrix, columns with singular values less than the target threshold value are taken as a layer null space basis matrix, and a layer sub-network weight matrix is constructed based on the layer null space basis matrix and each column of the layer network weight matrix; based on the layer sub-network weight matrix, determining an invalid feature sensitivity of each expert network of the target inference layer.

4. The method of claim 3, wherein, The obtaining of the target threshold value corresponding to the target inference layer comprises: based on a feature dimension extracted by the target inference layer in an inference process, determining the target threshold value, wherein the lower the feature dimension extracted by the target inference layer, the larger the target threshold value.

5. The method of claim 3, wherein, The obtaining of the target threshold value corresponding to the target inference layer comprises: obtaining a divergence between an input feature distribution and an output feature distribution of the target inference layer, the divergence being used to represent a difference degree between the input feature distribution and the output feature distribution; based on the divergence, determining the target threshold value, wherein the smaller the divergence, the larger the target threshold value.

6. The method of claim 3, wherein, The obtaining of the target threshold value corresponding to the target inference layer comprises: obtaining a first number of times that an expert network of the target inference layer is activated; obtaining a second number of times that an expert network of any other reasoning layer except the target reasoning layer is activated as a reference reasoning layer; if the second number of times is greater than the first number of times, the target threshold value is less than a threshold value of the reference reasoning layer, and if the second number of times is less than the first number of times, the target threshold value of the target reasoning layer is greater than the threshold value of the reference reasoning layer.

7. The method of claim 3, wherein, The method further comprises: obtaining a singular value distribution of a layer network weight matrix corresponding to the target reasoning layer; determining a difference value between any two adjacent singular values in the singular value distribution; determining the target threshold value based on volatility of the difference value, wherein the greater the volatility of the difference value, the smaller the target threshold value.

8. The method of claim 7, wherein, The method further comprises: if there is a difference value greater than a difference value threshold value, finding a maximum difference value and two adjacent singular values corresponding to the maximum difference value in the obtained difference value; in the two adjacent singular values found, a value less than or equal to a minimum singular value is taken as the target threshold value.

9. The method of claim 2, wherein, The method further comprises: obtaining a number of expert network pruning of the target reasoning layer; pruning the expert network of the target reasoning layer in order from high to low according to the invalid feature sensitivity, and taking the number of expert network pruning as a constraint.

10. The method of claim 9, wherein, The method further comprises: obtaining a total number of expert networks of the target reasoning layer and a pruning ratio, wherein the lower the feature dimension extracted by the target reasoning layer in the reasoning process, the lower the pruning ratio, or the more times the expert network of the target reasoning layer is activated, the lower the pruning ratio; determining the number of expert network pruning based on the total number of expert networks and the pruning ratio.

11. The method of claim 9, wherein, The method further comprises: finding a number of expert networks of the target reasoning layer whose invalid feature sensitivity exceeds a sensitivity threshold value in the expert network of the target reasoning layer; obtaining a divergence between an input feature distribution and an output feature distribution of the target reasoning layer, the divergence being used to represent a difference degree between the input feature distribution and the output feature distribution; determining the number of expert network pruning based on the divergence and the number of expert networks whose invalid feature sensitivity exceeds the sensitivity threshold value, wherein the smaller the divergence, the fewer the number of network pruning, and the more the number of expert networks whose invalid feature sensitivity exceeds the sensitivity threshold value, the fewer the number of network pruning.

12. The method of claim 9, wherein, The method further comprises: if the invalid feature sensitivity of each expert network of the target reasoning layer is lower than a sensitivity threshold value, the expert network of the target reasoning layer is not pruned.

13. The method of claim 9, wherein, After the network parameters of the mixed expert model are pruned, the method further comprises: deleting columns corresponding to the pruned expert network in the network weight matrix to obtain a new network weight matrix; obtaining domain sample data based on a target domain to which the mixed expert model needs to be applied. Based on the new network weight matrix and the domain sample data, the mixed expert model is fine-tuned to adjust the new network weight matrix and / or the classification head of the mixed expert model.

14. The method of claim 1, wherein, The sub-network weight matrix is constructed based on the null space basis matrix and each column of the network weight matrix, including: In the network weight matrix, any column is taken as a target column, which corresponds to a target expert network of the mixed expert model; The target column and the null space basis matrix are merged to obtain a sub-network weight matrix corresponding to the target expert network.

15. The method of claim 14, wherein, The invalid feature sensitivity of each expert network is determined based on the maximum singular value and the minimum singular value, including: The sub-network weight matrix corresponding to the target expert network is singular value decomposed to obtain a corresponding maximum singular value and a minimum singular value; The invalid feature sensitivity of the target expert network is determined based on the obtained maximum singular value and minimum singular value.

16. The method of claim 15, wherein, The invalid feature sensitivity of the target expert network is determined based on the obtained maximum singular value and minimum singular value, including: A mapping relationship between the maximum singular value, the minimum singular value and the invalid feature sensitivity is constructed, and the invalid feature sensitivity of the target expert network is obtained based on the mapping relationship, wherein in the mapping relationship, the maximum singular value is directly proportional to the invalid feature sensitivity, and the minimum singular value is inversely proportional to the invalid feature sensitivity.

17. A model processing apparatus characterized by comprising: The device is configured in an edge device, and the device includes: A weight matrix acquisition module is configured to acquire a network weight matrix of a mixed expert model, the mixed expert model including a plurality of expert networks, columns of the network weight matrix corresponding to the expert networks one by one, and each column being used to represent an activation mode of the corresponding expert network, wherein the mixed expert model is used for image processing or text processing; A first singular value decomposition module is configured to singular value decompose the network weight matrix to obtain a right singular vector matrix; A weight matrix reconstruction module is configured to, in the right singular vector matrix, take columns with singular values less than a threshold value as a null space basis matrix, and construct a sub-network weight matrix based on the null space basis matrix and each column of the network weight matrix; A second singular value decomposition module is configured to singular value decompose the sub-network weight matrix to obtain a maximum singular value and a minimum singular value; A parameter pruning module is configured to determine invalid feature sensitivity of each expert network based on the maximum singular value and the minimum singular value, and prune network parameters of the mixed expert model based on the invalid feature sensitivity to reduce the number of network parameters of the mixed expert model.

18. An electronic device, comprising: It includes: A memory is configured to store a computer program; A processor is configured to implement the model processing method of any one of claims 1-16 when executing the computer program.

19. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the model processing method of any one of claims 1-16. The computer readable storage medium stores a computer program, wherein the computer program is executed by the processor to implement the model processing method of any one of claims 1-16.

20. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the model processing method according to any one of claims 1 to 16.

Images