Multi-model compression method and device, task processing method and equipment and storage medium

Abstract

The invention relates to the technical field of model compression, and provides a multi-model compression method and device, a task processing method and equipment and a storage medium, and the multi-model compression method comprises the steps: taking a plurality of to-be-compressed models as leaf nodes, carrying out bottom-up hierarchical clustering on each leaf node based on weight similarity, and constructing a residual quantization tree; starting from a root node of the residual quantization tree, performing top-down residual quantization on the residual quantization tree to obtain quantization parameters of each leaf node; the quantization parameter comprises a quantization weight matrix of a leaf node obtained by performing quantization compression on a weight matrix of the to-be-compressed model, and a quantization coefficient of a target model allocated to the leaf node; the quantized weight matrix is shared by each target model. Through bottom-up hierarchical clustering and top-down residual quantization on the basis of clustering, quantitative compression of multiple models can be realized under the condition of not introducing large precision loss, and the accuracy of model reasoning is ensured.

Description

Technical Field

[0001] This invention relates to the field of model compression technology, and in particular to a multi-model compression method, apparatus, task processing method, device, and storage medium. Background Technology

[0002] Large Language Models (LLMs) have demonstrated remarkable capabilities in general language understanding. However, LLMs based on the Transformer architecture face the limitation that the computational cost of the attention mechanism increases quadratically with sequence length. Therefore, much research focuses on implementing more efficient attention variants, such as Structured State Space Models (SSMs) and linear attention. Meanwhile, these pre-trained large language models generally have a large number of parameters, necessitating model compression techniques to reduce their size and enable deployment on lightweight devices.

[0003] Currently, compression methods for large models typically involve lightweight deployments of individual models. However, in language model service scenarios, it is often necessary to deploy multiple models to run collaboratively, such as models for different downstream tasks and user-customized models. Compression methods for individual models are not suitable for scenarios involving multi-model deployments. In multi-model collaborative scenarios, compressing each model individually often results in significant accuracy loss, affecting the accuracy of inference results. Summary of the Invention

[0004] This invention provides a multi-model compression method, apparatus, task processing method, device, and storage medium to address the shortcomings of existing technologies where individual compression of each model in the context of multi-model collaboration results in significant accuracy loss and affects the accuracy of inference results.

[0005] This invention provides a multi-model compression method, comprising the following steps: Multiple models to be compressed are used as leaf nodes. Based on the weight similarity of each leaf node, a bottom-up hierarchical clustering is performed on each leaf node to construct a residual quantization tree. Starting from the root node of the residual quantization tree, perform top-down residual quantization on the residual quantization tree to obtain the quantization parameters of each leaf node; The quantization parameters include a quantization weight matrix and quantization coefficients of the target models assigned to the leaf nodes; the quantization weight matrix is ​​shared by each of the target models, and the quantization weight matrix and the quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0006] According to the multi-model compression method provided by the present invention, the step of performing bottom-up hierarchical clustering on each leaf node based on the weight similarity of each leaf node to construct a residual quantization tree includes: Based on the weight matrix of each leaf node, calculate the matrix distance between each pair of leaf nodes; By minimizing the sum of the matrix distances, non-crossing clustering is performed on each leaf node, the paired leaf nodes are merged into a parent node, and the node parameters of the parent node are generated. The parent node is used as a new leaf node. The step of calculating the matrix distance between each pair of leaf nodes based on the weight matrix of each leaf node is returned and executed until the merged parent node is the root node, thus obtaining the residual quantization tree.

[0007] According to the multi-model compression method provided by the present invention, the node parameters include a model index set and a child node set; the node parameters for generating the parent node include: Based on the target model contained in the leaf nodes of the synthesized parent node, a set of model indexes for the parent node is generated; Based on the node identifiers of the leaf nodes of the synthesized parent node, a set of child nodes of the parent node is generated.

[0008] According to the multi-model compression method provided by the present invention, the residual quantization is performed from top to bottom on the residual quantization tree, starting from the root node, to obtain the quantization parameters of each leaf node, including... Starting from the root node of the residual quantization tree, the root node is taken as the target node, and the objective function is constructed with minimizing the weight error of the target node as the optimization objective. The weight error is the error between the weight matrix of each target model contained in the target node and the optimization object. The optimization object is the product of the quantization coefficient of the target model and the quantization weight matrix of the target node. By employing an alternating optimization approach, the optimal solution for the optimization object is obtained based on the objective function, thereby acquiring the quantization parameters of the target node; The quantization parameters are passed down to the child nodes of the target node, and the child nodes are used as new target nodes. The process is repeated, and the objective function is constructed with minimizing the weight error of the target node as the optimization objective, until the target node is the last level leaf node in the residual quantization tree.

[0009] According to the multi-model compression method provided by the present invention, the step of employing alternating optimization to solve for the optimal solution of the optimization object based on the objective function, and obtaining the quantization parameters of the target node, includes: An alternating optimization approach is adopted, fixing the values ​​of the quantization weight matrix in the optimization object, and solving for the optimal solution of the quantization coefficients in the optimization object based on the objective function; With the values ​​of the quantization coefficients in the optimization object fixed, the optimal solution of the quantization weight matrix is ​​obtained based on the objective function; The quantization parameters of the target node are determined based on the optimal solution of the quantization weight matrix and the optimal solution of the quantization coefficients.

[0010] According to the multi-model compression method provided by the present invention, after performing top-down residual quantization on the residual quantization tree starting from the root node to obtain the quantization parameters of each leaf node, the method further includes: During the inference phase, the node paths of the inference model corresponding to the input features are determined; the node paths represent the paths from the root node to the leaf nodes corresponding to the inference model. Based on the same node on the node path, the product of the quantization weight matrix and the quantization coefficients of the inference model is calculated to obtain the quantization weight of the inference model; The input features are weighted according to the quantization weights to obtain the computation results of the inference model at each node of the node path; The calculation results at each node of the node path are accumulated to obtain the output feature corresponding to the input feature.

[0011] This invention also provides a task processing method, comprising: Convert the input text of the task to be processed into word vectors; The word vectors are input into the target large model for inference to obtain the processing result of the target large model on the task to be processed. The target large model is obtained by compressing multiple initial large models to be compressed using any of the multi-model compression methods described above.

[0012] The present invention also provides a multi-model compression device, comprising the following modules: The hierarchical clustering module is used to treat multiple models to be compressed as leaf nodes, and to perform bottom-up hierarchical clustering on each leaf node based on the weight similarity of each leaf node to construct a residual quantization tree. The quantization compression module is used to perform top-down residual quantization on the residual quantization tree, starting from the root node, to obtain the quantization parameters of each leaf node. The quantization parameters include a quantization weight matrix and quantization coefficients of the target models assigned to the leaf nodes; the quantization weight matrix is ​​shared by each of the target models, and the quantization weight matrix and the quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-model compression method as described above, or the task processing method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-model compression method as described above, or the task processing method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-model compression method as described above, or the task processing method as described above.

[0016] The multi-model compression method, apparatus, task processing method, device, and storage medium provided by this invention treats the models to be compressed as leaf nodes. Based on the weight similarity of the leaf nodes, a bottom-up hierarchical clustering is performed on the leaf nodes to construct a residual quantization tree. Starting from the root node, top-down residual quantization is performed on the residual quantization tree to obtain the quantization parameters of the leaf nodes. By hierarchically clustering the models to be compressed and performing residual quantization based on the clustering, collaborative quantization compression of multiple models can be achieved without introducing significant accuracy loss, ensuring the accuracy of model inference results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the multi-model compression method provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the hierarchical clustering process provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the weight quantization process provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the model reasoning process provided by the present invention.

[0022] Figure 5 This is a flowchart illustrating the task processing method provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the structure of the multi-model compression device provided by the present invention.

[0024] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] This invention provides a multi-model compression method that can compress multiple large language models with low precision loss in scenarios where multiple models operate collaboratively, thereby enabling lightweight deployment and application of multiple models.

[0027] Specifically, Figure 1 This is a flowchart illustrating the multi-model compression method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 100: Using multiple models to be compressed as leaf nodes, perform bottom-up hierarchical clustering on each leaf node based on the weight similarity of each leaf node to construct a residual quantization tree; Step 200: Starting from the root node of the residual quantization tree, perform top-down residual quantization on the residual quantization tree to obtain the quantization parameters of each leaf node. The quantization parameters include a quantization weight matrix and quantization coefficients of the target models assigned to the leaf nodes; the quantization weight matrix is ​​shared by each of the target models, and the quantization weight matrix and the quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0028] Multiple models to be compressed are used as leaf nodes. Based on the weight similarity of each leaf node, a bottom-up hierarchical clustering is performed on the leaf nodes to construct a residual quantization tree. Among them, the models to be compressed are large-scale language models (LLM), also known as large language models or large models.

[0029] The leaf nodes are clustered hierarchically from bottom to top. That is, the model to be compressed is taken as the lowest level leaf node. Based on the weight similarity of the leaf nodes, the leaf nodes at the same level are clustered. The leaf nodes in the same category obtained by clustering are merged into the nodes of the next level. The tree structure constructed in this way is the residual quantization tree.

[0030] In this model, the weight matrix of the leaf node is the weight matrix of the linear layer of the model to be compressed. The weight matrix of the model to be compressed is related to the task type it processes. Models to be compressed that process the same type of task have high similarity in their weight matrices, while models to be compressed that process different types of tasks have low similarity in their weight matrices. Clustering the leaf nodes based on weight similarity ensures that each node in the residual quantization tree contains models that process the same or similar types of tasks.

[0031] The tree structure constructed through hierarchical clustering is used to quantize the weight matrix of the model to be compressed; therefore, it is called a residual quantization tree. Starting from the root node of the residual quantization tree, top-down residual quantization is performed to obtain the quantization parameters of each leaf node. These quantization parameters include the quantization weight matrix and the quantization coefficients of the target model assigned to the leaf nodes. The quantization weight matrix and quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0032] That is, each model to be compressed is treated as a leaf node. Based on the weight similarity between leaf nodes, the leaf nodes are clustered hierarchically from bottom to top to construct a residual quantization tree. The residual quantization is then performed on the leaf nodes at each level from top to bottom according to the residual quantization tree. The weight matrix of the model to be compressed in each node is quantized into integer weights, so as to obtain the quantization coefficients of each model to be compressed and the quantization weight matrix of each node.

[0033] Top-down residual quantization of the residual quantization tree involves performing residual quantization on each node of the tree, resulting in the quantization weight matrix for each node and the quantization coefficients of the model to be compressed contained in that node. Optionally, the quantization coefficients of the same model to be compressed are generally different on different nodes, and nodes containing the same model to be compressed have a parent-child hierarchical relationship in the residual quantization tree.

[0034] Optionally, the quantization weight matrix of a node and the quantization coefficients of the model to be compressed on the node are obtained by performing residual quantization on the model to be compressed. The quantization weight matrix of a node and the quantization coefficients of the model to be compressed are used to quantize the weight matrix of the model to be compressed into integer weights. Models to be compressed located on the same node share the quantization weight matrix of that node.

[0035] In this embodiment, the model to be compressed is used as leaf nodes. Based on the weight similarity of the leaf nodes, a bottom-up hierarchical clustering is performed on the leaf nodes to construct a residual quantization tree. Starting from the root node, the residual quantization tree is performed from top to bottom to obtain the quantization parameters of the leaf nodes. By hierarchically clustering the model to be compressed and performing residual quantization based on the clustering, collaborative quantization compression of multiple models can be achieved without introducing significant accuracy loss, ensuring the accuracy of the model inference results.

[0036] In one embodiment, hierarchical clustering of leaf nodes is based on the weight similarity between leaf nodes, which is obtained by evaluating the distance between the weight matrices of the leaf nodes. Specifically, in step 100, based on the weight similarity of each leaf node, bottom-up hierarchical clustering is performed on each leaf node to construct a residual quantization tree, which specifically includes: Step 110: Calculate the matrix distance between each pair of leaf nodes based on the weight matrix of each leaf node; Step 120: By minimizing the sum of the matrix distances, perform non-crossing clustering pairing on each leaf node, merge the paired leaf nodes into a parent node, and generate the node parameters of the parent node; Step 130: Take the parent node as the new leaf node, return and execute the step of calculating the matrix distance between each pair of leaf nodes based on the weight matrix of each leaf node, until the merged parent node is the root node, and obtain the residual quantization tree.

[0037] Based on the weight matrix of each leaf node, the matrix distance between each pair of leaf nodes is calculated. By minimizing the sum of the matrix distances, the leaf nodes are clustered without intersection. The paired leaf nodes are merged into a parent node, and the node parameters of the parent node are generated. The merged parent node is used as a new leaf node, and the above steps of calculating the matrix distance between each pair of leaf nodes are repeated until the merged parent node becomes the root node, thus obtaining the residual quantization tree.

[0038] Optionally, for N (N is a positive integer greater than 1) models to be compressed, they are treated as N leaf nodes, denoted as... Each leaf node corresponds to a model to be compressed. Indicates the first The nth node, that is, the nth node The leaf nodes corresponding to the models to be compressed.

[0039] The leaf nodes are clustered and paired by minimizing the total distance between their corresponding weight matrices: (1) in, It is a binary variable used to represent the first... The node and the first Whether the nodes are paired Represents the weight matrix With weight matrix Matrix distance between them It is the first The weight matrix of the model to be compressed contained in each node. Indicates the first Each node contains the weight matrix of the model to be compressed.

[0040] Optionally, in this embodiment, the L2 norm is used as the distance metric between node weight distances. The L2 norm is also known as the Euclidean norm, and the weight matrix... With weight matrix The matrix distance between them is shown in Formula 2 below: (2) By minimizing the sum of matrix distances, we pair the nodes with the smallest matrix distances together to obtain the optimal pairing set. The paired node pairs are merged into their parent node using the method shown in Formula 3-5 below. : (3) (4) (5) Formula 3-5 illustrates the process of generating node parameters for a parent node. For a merged parent node, node parameters are generated based on the node parameters of its child nodes. The node parameters include a set of model indexes and a set of child nodes. The set of model indexes is used to characterize the target model contained in the node, and the set of child nodes is used to characterize each child node of the node.

[0041] Based on this, in step 120, the node parameters for generating the parent node further include: Step 121: Based on the target model contained in the leaf node of the synthesized parent node, generate a model index set for the parent node; Step 122: Generate a set of child nodes of the parent node based on the node identifiers of the leaf nodes of the synthesized parent node.

[0042] When generating the node parameters of the parent node, a set of model indexes for the parent node is generated based on the target models contained in the leaf nodes of the synthesized parent node. A set of child nodes for the parent node is generated based on the node identifiers of the leaf nodes of the synthesized parent node, which is used to index its child nodes.

[0043] Optionally, the node identifier of a leaf node is generated when the leaf node is generated. Each leaf node has a unique node identifier, which is used to distinguish different nodes.

[0044] In Equations 3-5, the weight matrix of the parent node is the average of the weight matrices of the leaf nodes merged into the parent node, as shown in Equation 3. Represents a node With nodes The weight matrix of the parent node is the node's weight matrix. With nodes The mean of the weight matrix, Represents a node With nodes The set of model indexes of the parent node, i.e. , , This represents the set of child nodes of a parent node, indicating the set of child nodes that the parent node contains. With nodes .

[0045] Following the method shown in Formulas 1-5 above, the model to be compressed is taken as a leaf node. Based on the weight similarity between leaf nodes, a bottom-up hierarchical clustering is performed on the leaf nodes. The weight matrix between leaf nodes is measured by distance to minimize the sum of distances between leaf nodes. The leaf nodes are clustered and paired, and the closest pair of leaf nodes is merged into a parent node. This includes, but is not limited to, generating a node identifier for the parent node, and using the average weight matrix of the leaf nodes as the weight matrix of the parent node.

[0046] Furthermore, based on the node parameters of the leaf nodes, the node parameters of the parent node are generated; specifically, based on the node identifiers of the leaf nodes, the set of child nodes of the parent node is generated. Based on the target models contained in the leaf nodes, generate a set of model indexes for the parent nodes. .

[0047] Finally, based on the weight matrices of each parent node, the merged parent node is used as the new leaf node. Based on the weight matrix of the new leaf node, the new leaf nodes are clustered and paired according to the similarity of the weight matrix until the root node is generated, thus obtaining the residual quantization tree.

[0048] Based on the constructed residual quantization tree, starting from the root node, perform top-down residual quantization to obtain the quantization parameters of the leaf nodes. Step 200 specifically includes: Step 210: Starting from the root node of the residual quantization tree, take the root node as the target node and construct an objective function with the goal of minimizing the weight error of the target node. The weight error is the sum of squares of the errors between the weight matrices of each target model contained in the target node and the optimization object. The optimization object is the product of the quantization coefficients of the target model and the quantization weight matrix of the target node. Step 220: Using an alternating optimization approach, the optimal solution of the optimization object is obtained based on the objective function to obtain the quantization parameters of the target node; Step 230: Pass the quantization parameters down to the child nodes of the target node, and use the child nodes as new target nodes. Return and execute the step of constructing the objective function with the goal of minimizing the weight error of the target node, until the target node is the last level leaf node in the residual quantization tree.

[0049] Starting from the root node of the residual quantization tree, the root node is taken as the target node. The objective function is constructed with the goal of minimizing the weight error of the target node. Here, the weight error is the error between the weight matrix of each target model contained in the target node and the optimization object, while the product of the quantization coefficient of the optimization object with respect to the target model and the quantization weight matrix of the target node is taken as the objective function.

[0050] Optionally, in this embodiment, the product of the quantization coefficients of the model and the quantization weight matrix of the node is used as the weight matrix in its inference stage to achieve quantization compression of the weight matrix, and the quantization residual is the error between the product of the quantization coefficients and the quantization weight matrix and the original weight matrix of the model.

[0051] The objective function is constructed based on the optimization object, which aims to minimize the weight error of the nodes. The optimization object includes the quantization weight matrix of the nodes and the quantization coefficients of each target model contained within the node. Specifically, when a leaf node contains multiple target models, the weight error of the node quantization is the sum of the squares of the errors of the weight matrices of each target model and the optimization object.

[0052] Since the optimization objects in the constructed objective function include the quantization weight matrix and quantization coefficients, an alternating optimization approach is adopted to solve for the optimal solutions of the optimization objects in the objective function. This includes the optimal solution for the quantization weight matrix and the optimal solution for the quantization coefficients of each objective model contained in the objective node, thus obtaining the quantization parameters of the objective node. Then, the quantization parameters of the objective node are passed down to its child nodes, and the child nodes are used as new objective nodes. The above steps of constructing the objective function with minimizing the weight error of the objective node as the optimization objective are repeated until the objective node is the last-level leaf node in the residual quantization tree.

[0053] In one embodiment, an alternating optimization approach is used to solve for the optimal solution of the optimization object based on the objective function. Specifically, the quantization coefficients in the optimization object are fixed, and the optimal solution of the quantization weight matrix is ​​solved. Then, the quantization weight matrix is ​​fixed again, and the optimal solution of the quantization coefficients is solved. Based on this, step 220, which uses an alternating optimization approach to solve for the optimal solution of the optimization object based on the objective function, includes: Step 221: Using an alternating optimization approach, the values ​​of the quantization weight matrix in the optimization object are fixed, and the optimal solution of the quantization coefficients in the optimization object is obtained based on the objective function; Step 222: Fix the values ​​of the quantization coefficients in the optimization object, and solve for the optimal solution of the quantization weight matrix based on the objective function; Step 223: Determine the quantization parameters of the target node based on the optimal solution of the quantization weight matrix and the optimal solution of the quantization coefficients.

[0054] An alternating optimization approach is adopted. The values ​​of the quantization weight matrix in the optimization object are fixed, and the optimal solution of the quantization coefficients in the optimization object is obtained based on the objective function. Then, the values ​​of the quantization coefficients in the optimization object are fixed, and the optimal solution of the quantization weight matrix is ​​obtained based on the objective function. Based on the optimal solution of the quantization weight matrix and the optimal solution of the quantization coefficients, the quantization parameters of the target node are determined.

[0055] Optionally, by using an alternating optimization approach, one can first fix the values ​​of the quantization coefficients in the optimization object, solve for the optimal solution of the quantization weight matrix based on the objective function, then fix the values ​​of the quantization weight matrix in the optimization object, solve for the optimal solution of the quantization coefficients based on the objective function, and determine the quantization parameters of the target node based on the optimal solutions of the quantization weight matrix and the quantization coefficients.

[0056] In one embodiment, the optimal quantization weight matrix for sharing is computed for each node by minimizing the mean square error of each node. and the quantization coefficients of the model to be compressed at this node The quantization weight matrix is ​​a binary integer matrix, and the quantization coefficients are used to characterize the quantization scale of the model to be compressed at the nodes.

[0057] Without loss of generality, assume and yes The vector in the vector, for the case of channel-by-channel quantization, In , Given the dimension of the input features of the model, the objective function is constructed as follows: (6) Since the quantization weight matrix of a node is shared among one or more models at that node, an alternating optimization approach is adopted, fixing the weight matrix each time. and Given one of them, find a closed-form solution for the other. Specifically, fix... The value is based on the objective function. right Find the partial derivative: (7) When the partial derivative is 0, we get The optimal solution can be obtained in different ways using this method. value corresponding The optimal solution.

[0058] Accordingly, fixed The value is obtained by solving for the weighted sum of the weight matrices of all models contained in the node. The optimal solution: (8) Finally, calculate the residual weights ( ), and pass it to the child nodes.

[0059] Based on this, if express The weight matrix of a linear layer in a model to be compressed and These represent the input and output dimensions of the linear layer, respectively. In the tree structure of the residual quantization tree, each node contains the following parameters: : Indicates that it has been assigned to a node The set of indexes for the model, i.e., the model index set; Used for all nodes assigned to them. The weight matrix of the model A shared quantization weight matrix for unified quantization; :node The set of quantization coefficients of the model, where, Corresponding to Medium model At the node Quantization coefficient; : Represents a node The set of child nodes that satisfy Furthermore, since the hierarchical clustering pairing of leaf nodes is performed without overlap, therefore, for any have .

[0060] In the residual quantization tree, each tree node independently stores the quantization coefficients for the models it contains. All models on a node share the low-bit quantization weight matrix of that node. The quantization weight of each model on a node is obtained by multiplying the quantization coefficient of the model with the quantization weight matrix of the node. The quantization weight of the same model on different nodes can be the same or different.

[0061] Based on this, after step 200, the following may also be included: Step 310: In the inference phase, determine the node path of the inference model corresponding to the input features; the node path represents the path from the root node to the leaf node corresponding to the inference model; Step 320: Based on the same node on the node path, calculate the product of the quantization weight matrix and the quantization coefficients of the inference model to obtain the quantization weights of the inference model; Step 330: The input features are weighted according to the quantization weights to obtain the computation results of the inference model at each node of the node path; Step 340: The calculation results at each node of the node path are accumulated to obtain the output feature corresponding to the input feature.

[0062] During the inference phase, the node paths of the inference model corresponding to the input features are determined. These node paths represent the path from the root node to the corresponding leaf node of the inference model. Based on the same node on the node path, the product of the quantization weight matrix and the quantization coefficients of the inference model is calculated, serving as the quantization weights of the inference model. The input features are then weighted according to the calculated quantization weights of the inference model to obtain the computation results of the inference model at each node on the node path. Finally, the computation results at each node on the node path are summed to obtain the output features corresponding to the input features, which is the inference result of the inference model for the input features.

[0063] Optionally, the quantization weights of the inference model at the nodes are weight matrices obtained by quantizing the quantization coefficients and the quantization weight matrix, as shown in the reference. Figure 2 The hierarchical clustering process shown, for the model to be compressed with N = 4, results in a residual quantization tree that is a 3-level binary tree. Figure 2 In the model 1 weight matrix Weight matrix of Model 3 The weight matrix of Model 2 has a high degree of similarity. The weight matrix of Model 4 Because they have high similarity, the leaf nodes corresponding to Model 1 and Model 3 are merged into a parent node, resulting in the left node in the second layer of leaf nodes. The leaf nodes corresponding to Model 2 and Model 4 are merged into a parent node, resulting in the right node in the second layer of leaf nodes. Finally, the leaf nodes of the second level are merged into the parent node to obtain the root node. ).

[0064] Furthermore, referring to Figure 3 The weight quantization process shown is as follows: Figure 2 Taking the left node of the second-level leaf nodes as an example, this leaf node contains the weight matrices of Model 1 and Model 3. and The quantization weights at this node are respectively the quantization weight matrix of this node. and their respective quantization coefficients and The product of these terms yields the quantization weights of Model 1 and Model 3 at the left node, respectively. and This storage method, which stores quantization coefficients and quantization weight matrices on nodes, exceeds the effective bit width of the storage bit width. It achieves the same effect as traditional model compression methods in terms of compression performance and storage overhead, thereby maximizing model compression with minimal loss of model accuracy.

[0065] The overall quantization weight of the model is the sum of the quantization weights of all nodes on the path from the root node to its corresponding last-level leaf node, that is: (9) in, Indicates from the root node to the node containing the first node. The node paths of the leaf nodes of each model. The quantized weights of the model at each node are summed with the input features to form the output features. Specifically, for the input features of different models... Output features It can be calculated as follows: (10) At the same node, different models share a single quantization weight matrix. Therefore, the quantitative inference of input features for different models, that is... The computation can be processed in parallel. For example... Figure 4 The model inference process shown applies to the input features of Model 1 and Model 3. and During top-down reasoning, Model 1 and Model 3 share common ancestor nodes, namely nodes A and B. The parent node of Model 1 is node C, and the parent node of Model 3 is node D. The quantization weights of Model 1 and Model 3 during reasoning are as follows: (11) (12) During the inference phase, Model 1 and Model 3 share the low-bit quantization weight matrices on nodes A and B for quantization inference. For Model 1, the node path is Node A-Node B-Node C-Node 1. Based on the quantization coefficients and quantization weight matrices of each node on the node path, Model 1 performs quantization inference on the input features at each node along the path, obtaining the computation results of each node. These computation results are then summed to obtain the output features of Model 1 for the input features, serving as the inference result of Model 1. Similarly, for Model 3, the node path is Node A-Node B-Node D-Node 3. Based on the quantization coefficients and quantization weight matrices of each node on the node path, Model 3 performs quantization inference on the input features at each node along the path, obtaining the computation results of each node. These computation results are then summed to obtain the output features of Model 3 for the input features, serving as the inference result of Model 3.

[0066] In this embodiment, by using the model to be compressed as leaf nodes, hierarchical clustering is performed from bottom to top based on weight similarity. Leaf nodes with similar weights are merged until a root node is generated, thus constructing a residual quantization tree. The residual quantization tree is then subjected to top-down residual quantization to obtain the quantization weight matrix of each node and the quantization coefficients of the model to be compressed at that node. The product of the quantization weight matrix and the quantization coefficients is used to replace the model's weight matrix, thereby achieving quantization compression of the model's weight matrix. This approach can maximize the quantization compression of multiple models while ensuring model accuracy, thus meeting the model compression requirements in multi-model deployment application scenarios.

[0067] Furthermore, in the hierarchical clustering of leaf nodes, the distance between weight matrices is used to measure the weight similarity between leaf nodes, and non-crossover clustering is performed on the leaf nodes to ensure that models with similar functions have the same parent and ancestor nodes, which helps to reduce quantization error. In the residual quantization process, the objective function is constructed by minimizing the weight error between the model's weight matrix and the optimization object. An alternating optimization approach is used to solve for the optimal solutions of the quantization weight matrix and the model's quantization coefficients, which serve as the quantization parameters for the nodes. Based on the clustering results of weight similarity, quantization compression of multiple models is achieved from the node dimension, ensuring that the quantization process does not introduce significant accuracy loss.

[0068] Based on the quantization compression of the model weight matrix, during the inference phase, the product of the quantized weight matrix and the quantization coefficients replaces the model's weight matrix. The quantized inference of the input features by all nodes along the path from the root node to the corresponding leaf node is summed and used as the model's output features, ensuring model performance. Furthermore, multiple models on the same node share the same low-bit quantization weight matrix, significantly increasing the effective bit width for storing model parameters. This achieves an optimal balance between compression performance and storage overhead, reducing storage costs.

[0069] Reference Figure 5 This invention also provides a task processing method, comprising the following steps: S1, converts the input text of the task to be processed into word vectors; S2, input the word vectors into the target large model for inference, and obtain the processing result of the target large model on the task to be processed; The target large model is obtained by compressing multiple initial large models to be compressed using the multi-model compression method described in the above embodiments.

[0070] For each task to be processed, the corresponding input text is converted into word vectors, and these word vectors are then input into the target large model for inference, yielding the processing result of the target large model for the task. The target large model is obtained by compressing multiple initial large models to be compressed using the multi-model compression method described in the above embodiments; the compression process of the initial large models will not be elaborated further here.

[0071] It should be noted that the multi-model compression method provided in the embodiments of the present invention is not only applicable to the compression of multiple large models, but also applicable to the compression of multiple expert sub-models in a large model. That is, multiple expert sub-models in a large model can be used as models to be compressed, and the multi-model compression method described in the above embodiments can be used to quantize and compress multiple expert sub-models to obtain the target large model.

[0072] In this embodiment, by utilizing a large, quantized and compressed model for task processing, the number of parameters involved in the model inference stage is reduced, thus lowering the computational overhead of model inference. Furthermore, the quantized and compressed multiple models share the same quantization weight matrix on the same node and can achieve parallel inference, improving task processing efficiency.

[0073] The multi-model compression apparatus provided by the present invention is described below. The multi-model compression apparatus described below and the multi-model compression method described above can be referred to in correspondence.

[0074] Reference Figure 6 This invention provides a multi-model compression device, comprising: The hierarchical clustering module 10 is used to take multiple models to be compressed as leaf nodes, and perform bottom-up hierarchical clustering on each leaf node based on the weight similarity of each leaf node to construct a residual quantization tree. The quantization compression module 20 is used to perform top-down residual quantization on the residual quantization tree, starting from the root node, to obtain the quantization parameters of each leaf node. The quantization parameters include a quantization weight matrix and quantization coefficients of the target models assigned to the leaf nodes; the quantization weight matrix is ​​shared by each of the target models, and the quantization weight matrix and the quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0075] In one embodiment, the hierarchical clustering module 10 is further configured to: Based on the weight matrix of each leaf node, calculate the matrix distance between each pair of leaf nodes; By minimizing the sum of the matrix distances, non-crossing clustering is performed on each leaf node, the paired leaf nodes are merged into a parent node, and the node parameters of the parent node are generated. The parent node is used as a new leaf node. The step of calculating the matrix distance between each pair of leaf nodes based on the weight matrix of each leaf node is returned and executed until the merged parent node is the root node, thus obtaining the residual quantization tree.

[0076] In one embodiment, the node parameters include a model index set and a child node set; the hierarchical clustering module 10 is further configured to: Based on the target model contained in the leaf nodes of the synthesized parent node, a set of model indexes for the parent node is generated; Based on the node identifiers of the leaf nodes of the synthesized parent node, a set of child nodes of the parent node is generated.

[0077] In one embodiment, the quantization compression module 20 is further configured to: Starting from the root node of the residual quantization tree, the root node is taken as the target node, and the objective function is constructed with minimizing the weight error of the target node as the optimization objective. The weight error is the error between the weight matrix of each target model contained in the target node and the optimization object. The optimization object is the product of the quantization coefficient of the target model and the quantization weight matrix of the target node. By employing an alternating optimization approach, the optimal solution for the optimization object is obtained based on the objective function, thereby acquiring the quantization parameters of the target node; The quantization parameters are passed down to the child nodes of the target node, and the child nodes are used as new target nodes. The process is repeated, and the objective function is constructed with minimizing the weight error of the target node as the optimization objective, until the target node is the last level leaf node in the residual quantization tree.

[0078] In one embodiment, the quantization compression module 20 is further configured to: An alternating optimization approach is adopted, fixing the values ​​of the quantization weight matrix in the optimization object, and solving for the optimal solution of the quantization coefficients in the optimization object based on the objective function; With the values ​​of the quantization coefficients in the optimization object fixed, the optimal solution of the quantization weight matrix is ​​obtained based on the objective function; The quantization parameters of the target node are determined based on the optimal solution of the quantization weight matrix and the optimal solution of the quantization coefficients.

[0079] In one embodiment, the multi-model compression apparatus further includes a quantization inference module, used for: During the inference phase, the node paths of the inference model corresponding to the input features are determined; the node paths represent the paths from the root node to the leaf nodes corresponding to the inference model. Based on the same node on the node path, the product of the quantization weight matrix and the quantization coefficients of the inference model is calculated to obtain the quantization weight of the inference model; The input features are weighted according to the quantization weights to obtain the computation results of the inference model at each node of the node path; The calculation results at each node of the node path are accumulated to obtain the output feature corresponding to the input feature.

[0080] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute steps of the multi-model compression method, such as: Multiple models to be compressed are used as leaf nodes. Based on the weight similarity of each leaf node, a bottom-up hierarchical clustering is performed on each leaf node to construct a residual quantization tree. Starting from the root node of the residual quantization tree, perform top-down residual quantization on the residual quantization tree to obtain the quantization parameters of each leaf node; The quantization parameters include a quantization weight matrix and quantization coefficients of the target models assigned to the leaf nodes; the quantization weight matrix is ​​shared by each of the target models, and the quantization weight matrix and the quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0081] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to perform the steps of the multi-model compression method provided by the above methods, for example including: Multiple models to be compressed are used as leaf nodes. Based on the weight similarity of each leaf node, a bottom-up hierarchical clustering is performed on each leaf node to construct a residual quantization tree. Starting from the root node of the residual quantization tree, perform top-down residual quantization on the residual quantization tree to obtain the quantization parameters of each leaf node; The quantization parameters include a quantization weight matrix and quantization coefficients of the target models assigned to the leaf nodes; the quantization weight matrix is ​​shared by each of the target models, and the quantization weight matrix and the quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-model compression method provided by the above methods, for example including: Multiple models to be compressed are used as leaf nodes. Based on the weight similarity of each leaf node, a bottom-up hierarchical clustering is performed on each leaf node to construct a residual quantization tree. Starting from the root node of the residual quantization tree, perform top-down residual quantization on the residual quantization tree to obtain the quantization parameters of each leaf node; The quantization parameters include a quantization weight matrix and quantization coefficients of the target models assigned to the leaf nodes; the quantization weight matrix is ​​shared by each of the target models, and the quantization weight matrix and the quantization coefficients are obtained by quantizing and compressing the weight matrix of the target model.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-model compression method, characterized in that, include: Multiple models to be compressed are used as leaf nodes. Based on the weight similarity of each leaf node, a bottom-up hierarchical clustering is performed on each leaf node to construct a residual quantization tree. Starting from the root node of the residual quantization tree, perform top-down residual quantization on the residual quantization tree to obtain the quantization parameters of each leaf node; The quantization parameters include the quantization weight matrix and the quantization coefficients of the target model assigned to the leaf nodes; The quantization weight matrix is ​​shared by all the target models, and the quantization weight matrix and the quantization coefficient are obtained by quantizing and compressing the weight matrix of the target model.

2. The multi-model compression method according to claim 1, characterized in that, The step of constructing a residual quantization tree by performing bottom-up hierarchical clustering of the leaf nodes based on their weight similarity includes: Based on the weight matrix of each leaf node, calculate the matrix distance between each pair of leaf nodes; By minimizing the sum of the matrix distances, non-crossing clustering is performed on each leaf node, the paired leaf nodes are merged into a parent node, and the node parameters of the parent node are generated. The parent node is used as a new leaf node. The step of calculating the matrix distance between each pair of leaf nodes based on the weight matrix of each leaf node is returned and executed until the merged parent node is the root node, thus obtaining the residual quantization tree.

3. The multi-model compression method according to claim 2, characterized in that, The node parameters include a model index set and a child node set; the node parameters for generating the parent node include: Based on the target model contained in the leaf nodes of the synthesized parent node, a set of model indexes for the parent node is generated; Based on the node identifiers of the leaf nodes of the parent node, a set of child nodes of the parent node is generated.

4. The multi-model compression method according to claim 1, characterized in that, Starting from the root node of the residual quantization tree, a top-down residual quantization process is performed on the residual quantization tree to obtain the quantization parameters of each leaf node, including... Starting from the root node of the residual quantization tree, the root node is taken as the target node, and the objective function is constructed with minimizing the weight error of the target node as the optimization objective. The weight error is the error between the weight matrix of each target model contained in the target node and the optimization object, wherein the optimization object is the product of the quantization coefficient of the target model and the quantization weight matrix of the target node; By employing an alternating optimization approach, the optimal solution for the optimization object is obtained based on the objective function, thereby acquiring the quantization parameters of the target node; The quantization parameters are passed down to the child nodes of the target node, and the child nodes are used as new target nodes. The process is repeated, and the objective function is constructed with minimizing the weight error of the target node as the optimization objective, until the target node is the last level leaf node in the residual quantization tree.

5. The multi-model compression method according to claim 4, characterized in that, The method employs alternating optimization to solve for the optimal solution of the optimization object based on the objective function, thereby obtaining the quantization parameters of the target node, including: An alternating optimization approach is adopted, fixing the values ​​of the quantization weight matrix in the optimization object, and solving for the optimal solution of the quantization coefficients in the optimization object based on the objective function; With the values ​​of the quantization coefficients in the optimization object fixed, the optimal solution of the quantization weight matrix is ​​obtained based on the objective function; The quantization parameters of the target node are determined based on the optimal solution of the quantization weight matrix and the optimal solution of the quantization coefficients.

6. The multi-model compression method according to claim 1, characterized in that, After performing top-down residual quantization on the residual quantization tree, starting from the root node, to obtain the quantization parameters of each leaf node, the method further includes: During the inference phase, the node paths of the inference model corresponding to the input features are determined; the node paths represent the paths from the root node to the leaf nodes corresponding to the inference model. Based on the same node on the node path, the product of the quantization weight matrix and the quantization coefficients of the inference model is calculated to obtain the quantization weight of the inference model; The input features are weighted according to the quantization weights to obtain the computation results of the inference model at each node of the node path; The calculation results at each node of the node path are accumulated to obtain the output feature corresponding to the input feature.

7. A task processing method, characterized in that, include: Convert the input text of the task to be processed into word vectors; The word vectors are input into the target large model for inference to obtain the processing result of the target large model on the task to be processed. The target large model is obtained by compressing multiple initial large models to be compressed using the multi-model compression method as described in any one of claims 1 to 6.

8. A multi-model compression device, characterized in that, include: The hierarchical clustering module is used to treat multiple models to be compressed as leaf nodes, and to perform bottom-up hierarchical clustering on each leaf node based on the weight similarity of each leaf node to construct a residual quantization tree. The quantization compression module is used to perform top-down residual quantization on the residual quantization tree, starting from the root node, to obtain the quantization parameters of each leaf node. The quantization parameters include the quantization weight matrix and the quantization coefficients of the target model assigned to the leaf nodes; The quantization weight matrix is ​​shared by all the target models, and the quantization weight matrix and the quantization coefficient are obtained by quantizing and compressing the weight matrix of the target model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-model compression method as described in any one of claims 1 to 6, or the task processing method as described in claim 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-model compression method as described in any one of claims 1 to 6, or the task processing method as described in claim 7.

Images