Model compression method, electronic equipment, storage medium and program product
By pruning processing in the processing layer of the large language model, processing nodes with less impact are reduced, the problem of low model processing efficiency is solved, and parameter call reduction and processing efficiency improvement are achieved.
Patent Information
- Application Number
- CN202510222610.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-27
- Publication Date
- 2025-03-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Large language models occupy a large amount of computing resources and memory resources during the inference process, resulting in low processing efficiency.
By reducing the processing nodes with less impact in each processing layer in the initial model, the node pruning rate of each processing layer is determined, and pruning is performed based on this rate to obtain the target model.
Reduces the number of parameters called by the model at runtime and improves the processing efficiency of the model.
Smart Images
Figure CN119721167A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a model compression method, an electronic device, a storage medium and a program product. Background Art
[0002] Large Language Models (LLMs) can perform a variety of complex natural language processing tasks such as text generation, machine translation, question-answering systems, sentiment analysis, etc. They can be applied in scenarios such as search engine improvement, intelligent customer service, automatic summary generation, and language understanding.
[0003] In the related art, after LLMs are trained, they can be deployed. The deployed LLMs need to occupy a large amount of computing resources and memory resources during the reasoning process. At the same time, a large number of parameters need to be called for reasoning, which makes the processing efficiency of the model low. Summary of the invention
[0004] The present application provides a model compression method, an electronic device, a storage medium and a program product to at least solve the problem of low model processing efficiency in related technologies.
[0005] In a first aspect, the present application provides a model compression method, comprising:
[0006] Acquire an initial model, wherein the initial model includes a plurality of processing layers, and each processing layer includes a plurality of initial processing nodes;
[0007] Determining a first importance corresponding to each processing layer;
[0008] Determine, according to the first importance corresponding to each processing layer, a node pruning rate corresponding to each processing layer, wherein the node pruning rate is a ratio of initial processing nodes reduced in the processing layer;
[0009] Determining at least one pruning node corresponding to each processing layer according to the node pruning rate corresponding to each processing layer;
[0010] According to at least one pruning node corresponding to each processing layer, a target model corresponding to the initial model is determined.
[0011] In a second aspect, the present application further provides a model compression device, including an acquisition module, a first determination module, a second determination module, a third determination module and a fourth determination module, wherein:
[0012] The acquisition module is used to acquire an initial model, wherein the initial model includes multiple processing layers, and each processing layer includes multiple initial processing nodes;
[0013] The first determination module is used to determine the first importance corresponding to each processing layer;
[0014] The second determination module is used to determine the node pruning rate corresponding to each processing layer according to the first importance corresponding to each processing layer, and the node pruning rate is the ratio of the initial processing nodes reduced in the processing layer;
[0015] The third determination module is used to determine at least one pruning node corresponding to each processing layer according to the node pruning rate corresponding to each processing layer;
[0016] The fourth determination module is used to determine the target model corresponding to the initial model according to the at least one pruning node corresponding to each processing layer.
[0017] In a third aspect, the present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned model compression methods when executing the computer program.
[0018] In a fourth aspect, the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program implements the steps of any of the above-mentioned model compression methods when executed by a processor.
[0019] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned model compression methods when executed by a processor.
[0020] Through the model compression method, electronic device, storage medium and program product provided by the present application, the initial model includes multiple processing layers, each processing layer includes multiple processing nodes, and the processing nodes with less impact in each processing layer can be reduced to reduce the number of parameters called when the model is running, thereby improving the processing efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0022] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application;
[0023] Figure 2 A schematic diagram of a flow chart of a model compression method provided in an embodiment of the present application;
[0024] Figure 3A schematic diagram of the structure of an initial model provided in an embodiment of the present application;
[0025] Figure 4 A flowchart of another model compression method provided in an embodiment of the present application;
[0026] Figure 5 A schematic diagram of a process for determining multiple intermediate pruning combinations provided in an embodiment of the present application;
[0027] Figure 6 A schematic diagram of a model comparison provided in an embodiment of the present application;
[0028] Figure 7 A schematic diagram of the structure of a model compression device provided in an embodiment of the present application;
[0029] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION
[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0031] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.
[0032] LLMs can perform a variety of complex natural language processing tasks such as text generation, machine translation, question-answering systems, sentiment analysis, etc., and can be applied in scenarios such as search engine improvement, intelligent customer service, automatic summary generation, and language understanding.
[0033] The Mixture of Experts (MoE) model is a type of LLMs. Based on the traditional Transformer structure, the MoE model replaces the dense feed-forward neural network (FFN) with a sparse MoE layer. The sparse MoE layer consists of multiple expert models (such as neural networks) and a gating network (Router). The MoE model can decompose the task to be processed into multiple subtasks, each of which is processed by a dedicated expert model, and the gating network dynamically distributes the input data to different expert models.
[0034] In the related art, after LLMs are trained, they can be deployed. The deployed LLMs need to occupy a large amount of computing resources and memory resources during the reasoning process. At the same time, a large number of parameters need to be called for reasoning, which makes the processing efficiency of the model low.
[0035] In the embodiments provided in the present application, the initial model includes multiple processing layers (for example, the initial model is an MoE model, and the processing layer is an expert model), and each processing layer includes multiple processing nodes (for example, the initial model is an MoE model, and the multiple processing nodes are multiple experts in the expert model). This can reduce the number of processing nodes with less impact in each processing layer, reduce the number of calling parameters, and improve the processing efficiency of the model.
[0036] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0037] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the model compression method depends, the specific application environment architecture or specific hardware architecture is described here.
[0038] Figure 1 This is a schematic diagram of an application scenario provided by the embodiment of this application. Figure 1 , including a processing device 101 and a terminal device 102.
[0039] The initial model may be a trained MoE model. After the initial model is deployed on the terminal device 102, data may be input into the initial model through the terminal device 101, and the initial model may process the input data to obtain output data.
[0040] The initial model may include multiple processing layers, each processing layer may include multiple initial processing nodes, and before the initial model is deployed, the processing device 101 may obtain the initial model and determine the node pruning rate corresponding to each processing layer, where the node pruning rate is the ratio of the initial processing nodes reduced in the processing layer. The processing device 101 may determine at least one pruning node corresponding to each processing layer according to the node pruning rate corresponding to each processing layer, and reduce the initial processing nodes in each processing layer in the initial model according to the at least one pruning node corresponding to each processing layer, so as to obtain the target model corresponding to the initial model.
[0041] After the processing device 101 determines the target model corresponding to the initial model, the target model can be deployed on the terminal device 102 , and the user can use the target model through the terminal device 102 .
[0042] Figure 2 The following is a flow chart of the model compression method provided in the embodiment of the present application. Figure 2 As shown, an embodiment of the present application provides a model compression method, and the method is described in detail as follows:
[0043] S201, obtaining an initial model.
[0044] The initial model may include multiple processing layers, and each processing layer may include multiple initial processing nodes.
[0045] In order to facilitate the understanding of this application, Figure 3 The initial model provided in the embodiments of the present application is described.
[0046] Figure 3 This is a schematic diagram of the structure of an initial model provided in the embodiment of the present application. Figure 3 , the initial model includes layer normalization (Layer Norm), masked self-attention mechanism (Masked Self-Attention), gating network (Router), feedforward neural network (FFN) and language model head (LM Head). Layer normalization is used to accelerate training and improve model stability, masked self-attention mechanism is used to capture dependencies in input sequences, gating network is used to dynamically select the most appropriate expert model according to input features, feedforward neural network includes 4 processing layers, and processing layers are used to process pending tasks. The language model head is used to generate the final output.
[0047] After the initial model receives the input data, it is processed through layer normalization and masked self-attention mechanism, and then the input data is decomposed into multiple subtasks. Each subtask is input into the corresponding processing layer, and each subtask is processed. After the processing results obtained by each processing layer are combined, the final output result is output from the language model head. In addition, each processing layer includes multiple processing nodes, and the subtasks in each processing layer need to be processed by multiple processing nodes to obtain the processing results.
[0048] It is worth noting that the number of initial processing nodes included in each processing layer may be the same or different.
[0049] For example, assume that there are three processing layers, namely processing layer 1, processing layer 2 and processing layer 3. Assume that processing layer 1 includes four initial processing nodes, namely processing node 11, processing node 12, processing node 13 and processing node 14. Processing layer 2 includes three initial processing nodes, namely processing node 21, processing node 22 and processing node 23. Processing layer 3 includes four initial processing nodes, namely processing node 31, processing node 32, processing node 33 and processing node 34.
[0050] S202: Determine the first importance corresponding to each processing layer.
[0051] For any processing layer, the importance of the processing layer can be determined by the importance corresponding to each initial processing node in the processing layer. If the importance of the initial processing node in the processing layer is higher, the importance of the processing layer is higher.
[0052] Specifically, the second importance of each initial processing node in the processing layer may be determined; and the sum of the second importance of each initial processing node may be determined as the first importance corresponding to the processing layer.
[0053] For example, assuming that processing layer 1 has a total of 4 initial processing nodes, namely processing node 11, processing node 12, processing node 13 and processing node 14, assuming that the second importance of processing node 11 is 0.2, the second importance of processing node 12 is 0.3, the second importance of processing node 13 is 0.1, and the second importance of processing node 14 is 0.4, then it can be determined that the first importance corresponding to the processing layer is 1.
[0054] In some possible embodiments, test data may be obtained for any processing node, and first output data corresponding to the initial model and second output data corresponding to the first model may be determined based on the test data, and the second importance may be determined based on the first output data and the second output data.
[0055] The first model may be a model obtained by removing processing nodes from the initial model.
[0056] In some possible embodiments, there may be multiple test data. Multiple test data may be obtained, and the first output data and the second output data corresponding to each test data may be determined according to the multiple test data; the difference between the first output data and the second output data corresponding to each test data may be determined as the first difference corresponding to each test data; and the sum of the first difference corresponding to each test data may be determined as the second importance. Please refer to the following formula.
[0057]
[0058] in, Used to indicate the second importance of the jth processing node of the i-th processing layer; , represents the i-th processing nodes, Used to indicate the first processing node of the i-th layer, Used to indicate the i-th processing nodes; A dataset representing multiple test data, represents the output of the initial model including all processing nodes, Indicates to remove the processing node The output of the first model. represents the Frobenius norm, Used to indicate summing the first difference values of multiple test data.
[0059] i is an integer greater than or equal to 1 and less than or equal to L, where L is the number of the plurality of processing layers.
[0060] For any processing layer, the first importance can be determined by the following formula:
[0061]
[0062] in, Used to indicate the first importance of the i-th processing layer, It is used to indicate the second importance of the jth processing node of the i-th processing layer. The i-th processing layer has processing nodes.
[0063] S203: Determine the node pruning rate corresponding to each processing layer according to the first importance corresponding to each processing layer.
[0064] The node pruning rate may be a ratio of the initial processing nodes reduced in the processing layer.
[0065] For example, assuming that processing layer 1 includes 4 initial processing nodes, namely processing node 11, processing node 12, processing node 13 and processing node 14; assuming that the node pruning rate of processing layer 1 is 20%, any one of processing node 11, processing node 12, processing node 13 and processing node 14 can be pruned.
[0066] In the embodiment of the present application, the number of processing nodes to be removed from each processing layer can be determined by the first importance of each processing layer. If the importance of the processing layer is higher, fewer processing nodes are removed and more processing nodes are retained; if the importance of the processing layer is lower, more processing nodes are removed and fewer processing nodes are retained. The processing efficiency of the model can be improved by pruning while retaining the accuracy of the model.
[0067] S204: Determine at least one pruning node corresponding to each processing layer according to the node pruning rate corresponding to each processing layer.
[0068] For any processing layer, the number of pruned nodes may be determined according to the node pruning rate corresponding to the processing layer, and at least one corresponding pruned node may be determined from the multiple initial processing nodes according to the number of pruned nodes.
[0069] Pruning nodes are processing nodes that can be ignored when the initial model processes input data. Ignoring some pruning nodes that have little impact on data processing can improve the processing efficiency of the model during processing.
[0070] Specifically, multiple intermediate pruning combinations corresponding to each processing layer can be determined based on the node pruning rate corresponding to each processing layer; at least one processing layer set corresponding to the multiple processing layers can be determined; and at least one pruning node corresponding to each processing layer can be determined based on the multiple intermediate pruning combinations corresponding to each processing layer and at least one processing layer set.
[0071] The intermediate pruning combination may include at least one initial processing node.
[0072] For example, assuming that processing layer 1 has a total of 4 initial processing nodes, namely processing node 11, processing node 12, processing node 13 and processing node 14, assuming that the node pruning rate of processing layer 1 is 20%, it can be determined that the number of pruned nodes is 1, then 4 intermediate pruning combinations can be determined, intermediate pruning combination 1 includes 1 initial processing node, processing node 11; intermediate pruning combination 2 includes 1 initial processing node, processing node 12; intermediate pruning combination 3 includes 1 initial processing node, processing node 13; intermediate pruning combination 4 includes 1 initial processing node, processing node 14. Assuming that the node pruning rate of processing layer 1 is 50%, it can be determined that the number of pruned nodes is 2, then 6 intermediate pruning combinations can be determined, namely intermediate pruning combinations 1-6, wherein each intermediate pruning combination includes 2 initial processing nodes, as shown in Table 1.
[0073] Table 1
[0074]
[0075] Multiple processing layers can be divided into at least one processing layer set. For example, assuming there are 4 processing layers, namely processing layers 1-4, it can be determined that there are 2 processing layer sets, namely processing layer set 1 and processing layer set 2, wherein processing layer set 1 includes processing layer 1 and processing layer 2, and processing layer set 2 includes processing layer 3 and processing layer 4.
[0076] By processing a set of layers, multiple intermediate pruning combinations in each processing layer can be traversed to determine the pruning combination with the least impact in each processing layer, thereby improving the traversal efficiency and the processing efficiency of model compression.
[0077] S205. Determine a target model corresponding to the initial model according to at least one pruning node corresponding to each processing layer.
[0078] Specifically, according to at least one pruning node corresponding to each processing layer, a plurality of initial processing nodes corresponding to each processing layer are pruned to obtain a pruned model corresponding to the initial model; according to at least one target processing node corresponding to each processing layer, a decomposition model corresponding to the pruned model is determined; and by fine-tuning the model, the decomposition model is fine-tuned to obtain a target model.
[0079] The pruned model may be a model obtained by pruning at least one pruning node of each processing layer in the initial model, and the pruned model may include at least one target processing node corresponding to each processing layer.
[0080] Each processing node may also include a parameter matrix, where the parameter matrix includes a plurality of parameters that are called when the processing node processes the task to be processed.
[0081] The decomposition model may be a model obtained by performing low-rank decomposition on the initial parameter matrix of each target processing node corresponding to each processing layer in the pruning model.
[0082] For the performance degradation caused by pruning and low-rank decomposition, the efficient fine-tuning technology LoRA can be used for fine-tuning to update the weights of the decomposition model and obtain the target model.
[0083] In the embodiments provided in the present application, the node pruning rate corresponding to each processing layer can be determined based on the first importance of each processing layer in the initial model. Based on the node pruning rate corresponding to each processing layer, at least one pruning node corresponding to each processing layer can be determined. The initial model is pruned to obtain a target model. After the pruning process, the processing nodes in each processing layer that have a smaller impact on data processing can be reduced, thereby reducing the number of calling parameters and improving the processing efficiency of the model.
[0084] Figure 4 A flowchart of another model compression method provided in an embodiment of the present application. Figure 4 , the method may include:
[0085] S401: Obtain an initial model.
[0086] S402: Determine the first importance corresponding to each processing layer.
[0087] The execution process of S401-S402 can refer to the execution process of S201-S202, which will not be repeated here.
[0088] S403: Obtain a preset pruning rate.
[0089] In some possible embodiments, the preset pruning rate may be 20%.
[0090] The node pruning rate of each processing layer can be adjusted based on the preset pruning rate.
[0091] S404: Determine the node pruning rate corresponding to each processing layer according to the preset pruning rate and the first importance corresponding to each processing layer.
[0092] In some possible embodiments, the first importance corresponding to each processing layer is normalized to obtain a normalized importance corresponding to each processing layer; a pruning adjustment ratio corresponding to each processing layer is determined based on the normalized importance corresponding to each processing layer; and for any processing layer, the product of the pruning adjustment ratio corresponding to the processing layer and the preset pruning rate is determined as the node pruning rate corresponding to the processing layer.
[0093] The first importance of each processing layer can be normalized by referring to the following formula:
[0094]
[0095] in, Used to indicate the normalized importance of the i-th processing layer, Used to indicate the first importance of the i-th processing layer, and there are L processing layers in total.
[0096] Specifically, refer to the following formula to determine the node pruning rate:
[0097]
[0098] in, Used to indicate the node pruning rate of the i-th layer, Used to indicate the pruning adjustment ratio; Used to indicate the preset pruning rate.
[0099] Pruning adjustment ratio It can be determined according to the following formula:
[0100]
[0101] in, Used to indicate the normalized importance of the i-th processing layer, there are L processing layers in total.
[0102] S405: Determine multiple intermediate pruning combinations corresponding to each processing layer according to the node pruning rate corresponding to each processing layer.
[0103] For ease of understanding, below, combined Figure 5 , the execution process of determining multiple intermediate pruning combinations for any processing layer provided in the embodiment of the present application is described.
[0104] Figure 5 A schematic diagram of a process for determining multiple intermediate pruning combinations provided in an embodiment of the present application. Figure 5 , the method may include:
[0105] S501: Determine the number of pruned nodes according to the node pruning rate and the number of initial processing nodes corresponding to the processing layer.
[0106] In some possible embodiments, the number of pruned nodes may be determined based on the product of the node pruning rate and the number of initial processing nodes corresponding to the processing layer.
[0107] It is worth noting that the product of the node pruning rate and the number of initially processed nodes may be a decimal, and the number of pruned nodes can be determined by rounding. For example, assuming that the node pruning rate is 10% and the number of initially processed nodes is 1, the number of pruned nodes can be determined to be 1.
[0108] In some possible embodiments, if the number of pruned nodes is 0, the number of pruned nodes is determined to be 1 to ensure that at least one initial processing node is subtracted from each processing layer.
[0109] S502: Determine N initial pruning combinations from a plurality of initial processing nodes in the processing layer according to the number of pruning nodes.
[0110] Each initial pruning combination includes M initial processing nodes, where M is the number of pruning nodes, N is an integer greater than or equal to 1, and M is an integer greater than or equal to 1.
[0111] The M initial processing nodes may be any M of the multiple initial processing nodes of the processing layer.
[0112] For example, assuming that processing layer 1 has 10 initial processing nodes and the number of pruned nodes is 2, we can determine , that is, 45 initial pruning combinations.
[0113] In some possible embodiments, if the number N of the multiple initial pruning combinations is greater than the preset number , then determine For example, assuming the preset number is 50. Assuming that processing layer 1 has 10 initial processing nodes and the number of pruned nodes is 3, it can be determined , that is, 120 pruning combinations, 50 pruning combinations can be randomly determined from the 120 pruning combinations as the initial pruning combinations.
[0114] S503: Optimize the N initial pruning combinations to obtain multiple intermediate pruning combinations corresponding to the processing layer.
[0115] Specifically, a first loss difference corresponding to each of the N initial pruning combinations can be determined; an initial pruning combination whose first loss difference is less than a preset difference is determined as a combination to be optimized; and the combination to be optimized is iteratively optimized to obtain multiple intermediate pruning combinations.
[0116] The first loss difference is used to indicate the difference in output data loss after removing the initial processing nodes of the initial pruning combination in the initial model.
[0117] Multiple test data can be obtained to determine the second difference between the third output data corresponding to each test data and the first output data, and the sum of the norms of the second difference corresponding to each test data is determined as the first loss difference, wherein the third output data is the output data of the second model in the initial module after removing the initial processing nodes of the initial pruning combination.
[0118] The first loss difference can be determined by referring to the following formula:
[0119]
[0120] in, The first loss difference used to indicate the nth initial pruning combination of the i-th processing layer, for indicating the first output data of the initial model, The third output data of the second model is used to indicate the initial processing nodes of the initial model without the nth initial pruning combination of the i-th processing layer. A dataset representing multiple test data, represents the Frobenius norm, Used to indicate the sum of test results corresponding to multiple test data.
[0121] It is worth noting that the smaller the first loss difference is, the smaller the influence of the removed processing node in the initial model is, and it can be used as a pruning object.
[0122] An initial pruning combination whose first loss difference is smaller than a preset difference may be determined as a combination to be optimized.
[0123] For example, assuming that there are 50 initial pruning combinations in the i-th processing layer, assuming that the preset difference is 0.5, and assuming that the first loss difference of 30 initial pruning combinations is less than 0.5, then the 30 initial pruning combinations can be determined as the combinations to be optimized.
[0124] The combination to be optimized can be iteratively optimized by a genetic algorithm to obtain multiple intermediate pruning combinations, wherein the combination to be optimized is determined as the parent of the genetic algorithm, and multiple intermediate pruning combinations can be determined after the combination to be optimized is iterated multiple times by the genetic algorithm.
[0125] The maximum number of iterations of the genetic algorithm may be preset, for example, the maximum number of iterations may be 50; in addition, the number of multiple intermediate pruning combinations may also be preset, for example, the number may be 20 intermediate pruning combinations.
[0126] In the present application, the third output data after pruning the initial processing nodes corresponding to the initial pruning combination in the initial model can be compared with the first output data of the initial model to determine the first loss value corresponding to the initial pruning combination, and the initial pruning combination with less impact on the model can be screened by the first loss value, determined as the combination to be optimized, and then the combination to be optimized is optimized to obtain an intermediate pruning combination, which can improve the determination of the pruning combination with less impact on model processing in each processing layer, and after the model is compressed, the model can ensure the validity of the processed data.
[0127] S406. Determine at least one processing layer set corresponding to the multiple processing layers.
[0128] If the number of processing layers in each processing layer set is greater, the number of pruning combinations between processing layers determined in each processing layer set is greater, and the efficiency of determining at least one pruning node corresponding to each processing layer is lower. If the number of processing layers in each processing layer set is smaller, the efficiency of determining at least one pruning node corresponding to each processing layer can be improved, but the consideration of the impact of at least one pruning node corresponding to each processing layer on other processing layers is reduced, thereby reducing the effectiveness of determining at least one pruning node corresponding to each processing layer. Therefore, when determining the number of processing layers included in each processing layer set, it can be determined according to actual conditions, and no specific restrictions are made at this time.
[0129] S407: Determine at least one pruning node corresponding to each processing layer according to the multiple intermediate pruning combinations corresponding to each processing layer and at least one processing layer set.
[0130] In some possible embodiments, a target pruning combination corresponding to each processing layer can be determined based on multiple intermediate pruning combinations corresponding to each processing layer and at least one processing layer set; and at least one pruning node corresponding to each processing layer can be determined based on the target pruning combination corresponding to each processing layer.
[0131] The processing layer set can include T processing layers, where T is an integer greater than or equal to 1; there are L processing layers in the initial model, and the number of at least one processing layer set is , L is an integer greater than or equal to 1.
[0132] Then, for any processing layer set, multiple pruning sets corresponding to the processing layer set can be determined; the second loss difference corresponding to each pruning set can be determined; and the T intermediate pruning combinations corresponding to the pruning set with the smallest second loss difference are determined as the target pruning combinations corresponding to each processing layer in the processing layer set.
[0133] The pruning set may include T intermediate pruning combinations corresponding to the T processing layers.
[0134] For example, assuming that the processing layer set includes two processing layers, namely processing layer 1 and processing layer 2, assuming that processing layer 1 corresponds to two intermediate pruning combinations, namely pruning combination 11 and pruning combination 12, assuming that processing layer 2 corresponds to three intermediate pruning combinations, namely pruning combination 21, pruning combination 22 and pruning combination 23, then six pruning sets can be determined, and the pruning combinations corresponding to each pruning set can be shown in Table 2.
[0135] Table 2
[0136]
[0137] The second loss difference is used to indicate the value of the output data loss after removing the initial processing nodes corresponding to the pruning set in the initial model.
[0138] Multiple test data can be obtained to determine the third difference between the fourth output data corresponding to each test data and the first output data, and the sum of the norms of the third differences corresponding to each test data is determined as the second loss difference, wherein the fourth output data is the output data of the third model after removing the initial processing nodes corresponding to the pruning set in the initial module.
[0139] For example, assuming that the pruning set is pruning set 6, pruning set 6 includes pruning combination 12 and pruning combination 23, pruning combination 12 corresponds to the first and second processing nodes of the first processing layer, and pruning combination 23 corresponds to the first and fourth processing nodes of the second processing layer. Then in the initial model, these four processing nodes are removed, the test data is processed, and the fourth output data corresponding to the test data is obtained.
[0140] Furthermore, the second loss difference corresponding to each pruning set can be determined, and the pruning combination with the smallest second loss difference is the combination with the smallest impact between each processing layer in the processing layer set. The T intermediate pruning combinations corresponding to the pruning set can be determined as the target pruning combinations corresponding to each processing layer.
[0141] For example, assuming that as shown in Table 2, there are 6 pruning sets in processing layer set 1, among which the second loss difference corresponding to pruning set 3 is the smallest, then pruning combination 11 is determined as the target pruning combination corresponding to the first processing layer, and pruning combination 23 is determined as the target pruning combination corresponding to the second processing layer.
[0142] After the target pruning combination corresponding to each processing layer is determined, the initial processing node corresponding to the target pruning combination may be determined as at least one pruning node corresponding to each processing layer.
[0143] S408. Perform pruning processing on multiple initial processing nodes corresponding to each processing layer according to at least one pruning node corresponding to each processing layer to obtain a pruning model corresponding to the initial model.
[0144] The pruning model may include at least one target processing node corresponding to each processing layer.
[0145] Figure 6 A schematic diagram of a model comparison provided in the embodiment of the present application. Figure 6, the left side shows the L processing layers of the initial model and the N processing nodes corresponding to each processing layer. After pruning, the right side shows the L processing layers of the pruned model and the target processing nodes corresponding to each processing layer, where pruned processing nodes exist in each processing layer. Gray is used to indicate processing nodes that exist in the model, and white is used to indicate processing nodes that are pruned in the model.
[0146] S409: Determine a decomposition model corresponding to the pruning model according to at least one target processing node corresponding to each processing layer.
[0147] The pruned model can be decomposed through low-rank decomposition.
[0148] Low-rank decomposition is to decompose the original high-dimensional matrix into the product of two or more low-dimensional matrices. The matrix W is decomposed into ,in, yes The matrix of yes The matrix of In this way, the number of parameters is reduced from Reduce to , thus achieving compression. For example, when m=1000, n=1000, r=10, the number of parameters is reduced from 1000000 to 20000, and the compression rate can reach 95%. is an integer greater than or equal to 1, is an integer greater than or equal to 1, is an integer greater than or equal to 1.
[0149] Specifically, for any processing layer, the target decomposition rank corresponding to each target processing node can be determined; according to the target decomposition rank corresponding to each target processing node, the initial parameter matrix corresponding to each target processing node is decomposed to obtain multiple target parameter matrices corresponding to each target processing node; the initial parameter matrix of each target processing node corresponding to each processing layer in the pruning model is updated to multiple target parameter matrices to obtain a decomposition model.
[0150] The target decomposition rank is the rank r when the initial parameter matrix corresponding to the target processing node is decomposed.
[0151] In some possible embodiments, the initial parameter matrix may be decomposed by a singular value decomposition (SVD) method to obtain multiple target parameter matrices corresponding to each target processing node.
[0152] In some possible embodiments, the target decomposition rank corresponding to the target processing node can be determined according to the following method: obtaining a preset compression rate; determining the average rank corresponding to the pruning model according to the preset compression rate; determining the target decomposition rank according to the average rank and the second importance corresponding to each target processing node.
[0153] For any target processing node, the number of rows and columns of the initial parameter matrix corresponding to the target processing node can be obtained, and the first rank corresponding to the target processing node can be determined according to the preset compression rate, the number of rows and the number of columns. The average value of the first rank corresponding to each target processing node is determined as the average rank.
[0154] The first rank can be determined by referring to the following formula:
[0155]
[0156] in, Used to indicate the first rank, Used to indicate the preset compression ratio. Used to indicate the number of rows in the initial parameter matrix, Used to indicate the number of columns in the initial parameter matrix.
[0157] S410, fine-tuning the decomposition model by fine-tuning the model to obtain a target model.
[0158] To alleviate the performance degradation caused by pruning and low-rank decomposition, efficient fine-tuning technology (Low-Rank Adaptation, LoRA) can be used to perform fine-tuning, update the weights of the decomposition model, and obtain the target model.
[0159] The fine-tuning model may be a LoRA model, wherein the LoRA model may approximately simulate the parameter update amount during full fine-tuning by introducing a low-rank decomposition matrix into the weight matrix of the decomposition model, thereby reducing the number of parameters to be trained. Since the decomposition model has been low-rank decomposed in this application, when weight fine-tuning is performed through the LoRA model, the target parameter matrix after low-rank decomposition may be used.
[0160] The model compression method provided in the embodiment of the present application can determine the first importance of each processing layer according to the second importance of each processing node, and then determine the node pruning rate of each processing layer, so that the processing nodes with high importance can be retained and the processing nodes with low importance can be removed, thereby improving the effectiveness of model compression. In addition, by optimizing multiple initial pruning combinations, multiple intermediate pruning combinations corresponding to the processing layer can be obtained, and the target pruning combination can be selected from the multiple intermediate pruning combinations corresponding to the processing layer, so as to ensure the accuracy of model processing after pruning. By reducing the processing nodes with less influence in each processing layer, the number of calling parameters can be reduced, and the processing efficiency of the model can be improved.
[0161] It is worth noting that fine-tuning refers to further training on a task-specific dataset based on the pre-trained model to adjust the model parameters to better suit the target task. During fine-tuning, most of the layers of the pre-trained model are usually frozen, and only the newly added layers are trained or a small number of key layers are adjusted. This can not only retain the useful features learned by the pre-trained model, but also quickly adapt to the specific needs of the new task. In addition, choosing the right learning rate and training rounds is also the key to successful fine-tuning.
[0162] In the embodiment of the present application, the initial model is a pre-trained model, which refers to designing a language model training task based on a large-scale corpus (including language training materials such as sentences and paragraphs), training a large-scale neural network algorithm structure to learn and implement, and the large-scale neural network algorithm structure and parameters finally obtained are pre-trained models. Subsequent tasks can be based on the model to extract features or fine-tune tasks to achieve specific task purposes. The idea of pre-training is to first train a task to obtain a set of model parameters, then use the set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain models adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and can extract rich syntactic and semantic information from the text. The pre-trained model can provide tokens containing rich semantic information and sentence-level features for downstream tasks, and can also be fine-tuned directly on the pre-trained model for downstream tasks, so as to obtain downstream exclusive models quickly and conveniently.
[0163] Pre-training is a strategy for deep learning model training. Its core is to use large-scale data sets to preliminarily train the model so that the model can learn common feature representations. This process is similar to the basic learning stage of humans before learning new knowledge, accumulating experience through extensive reading and observation.
[0164] The neural network algorithm structure trained by the pre-trained model can be CNN, RNN, LSTM, etc., or a model built with an attention network, such as transformer, BERT, GPT, Clip, etc., which is not limited in this application. An attention network refers to a network model trained using an attention mechanism, which assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence, so that the model ultimately obtains a more accurate output.
[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.
[0166] Figure 7 This is a schematic diagram of the structure of the model compression device provided in the embodiment of the present application. Figure 7 The model compression device 700 may include an acquisition module 701, a first determination module 702, a second determination module 703, a third determination module 704 and a fourth determination module 705, wherein:
[0167] The acquisition module 701 is used to acquire an initial model, where the initial model includes multiple processing layers, and each processing layer includes multiple initial processing nodes;
[0168] The first determination module 702 is used to determine the first importance corresponding to each processing layer;
[0169] The second determination module 703 is used to determine the node pruning rate corresponding to each processing layer according to the first importance corresponding to each processing layer, and the node pruning rate is the ratio of the initial processing nodes reduced in the processing layer;
[0170] The third determination module 704 is used to determine at least one pruning node corresponding to each processing layer according to the node pruning rate corresponding to each processing layer;
[0171] The fourth determination module 705 is used to determine the target model corresponding to the initial model according to at least one pruning node corresponding to each processing layer.
[0172] In some possible embodiments, the second determining module 703 is specifically configured to:
[0173] Get the preset pruning rate;
[0174] The node pruning rate corresponding to each processing layer is determined according to the preset pruning rate and the first importance corresponding to each processing layer.
[0175] In some possible embodiments, the second determining module 703 is specifically configured to:
[0176] Normalizing the first importance corresponding to each processing layer to obtain a normalized importance corresponding to each processing layer;
[0177] According to the normalized importance of each processing layer, the pruning adjustment ratio corresponding to each processing layer is determined;
[0178] For any processing layer, the product of the pruning adjustment ratio corresponding to the processing layer and the preset pruning rate is determined as the node pruning rate corresponding to the processing layer.
[0179] In some possible embodiments, for any processing layer, the first determination module 702 is specifically used to:
[0180] determining a second importance of each initial processing node in the processing layer;
[0181] The sum of the second importance of each initial processing node is determined as the first importance corresponding to the processing layer.
[0182] In some possible embodiments, the third determining module 704 is specifically configured to:
[0183] Determine, according to the node pruning rate corresponding to each processing layer, a plurality of intermediate pruning combinations corresponding to each processing layer, wherein the intermediate pruning combination includes at least one initial processing node;
[0184] determining at least one processing layer set corresponding to the plurality of processing layers;
[0185] At least one pruning node corresponding to each processing layer is determined according to a plurality of intermediate pruning combinations corresponding to each processing layer and at least one processing layer set.
[0186] In some possible embodiments, for any processing layer, the third determination module 704 is specifically used to:
[0187] Determine the number of pruned nodes according to the node pruning rate and the number of initial processing nodes corresponding to the processing layer;
[0188] According to the number of pruned nodes, determine N initial pruning combinations from multiple initial processing nodes of the processing layer, each initial pruning combination includes M initial processing nodes, M is the number of pruned nodes, N is an integer greater than or equal to 1, and M is an integer greater than or equal to 1;
[0189] The N initial pruning combinations are optimized to obtain multiple intermediate pruning combinations corresponding to the processing layer.
[0190] In some possible embodiments, the third determining module 704 is specifically configured to:
[0191] Determine a first loss difference corresponding to each of the N initial pruning combinations, where the first loss difference is used to indicate a difference between second output data after removing an initial processing node of the initial pruning combination in the initial model and second output data of the initial module;
[0192] An initial pruning combination whose first loss difference is less than a preset difference is determined as a combination to be optimized;
[0193] The combination to be optimized is iteratively optimized to obtain multiple intermediate pruning combinations.
[0194] In some possible embodiments, the third determining module 704 is specifically configured to:
[0195] Determine a target pruning combination corresponding to each processing layer according to a plurality of intermediate pruning combinations corresponding to each processing layer and at least one processing layer set;
[0196] At least one pruning node corresponding to each processing layer is determined according to the target pruning combination corresponding to each processing layer.
[0197] In some possible embodiments, the processing layer set includes T processing layers, where T is an integer greater than or equal to 1; the initial model has a total of L processing layers, and the number of at least one processing layer set is , L is an integer greater than or equal to 1;
[0198] Wherein, for any processing layer set, the third determination module 704 is specifically used to:
[0199] Determine a plurality of pruning sets corresponding to the processing layer set, wherein the pruning sets include T intermediate pruning combinations corresponding to the T processing layers;
[0200] Determine a second loss difference corresponding to each pruning set;
[0201] The T intermediate pruning combinations corresponding to the pruning set with the smallest second loss difference are determined as the target pruning combination corresponding to each processing layer in the processing layer set.
[0202] In some possible embodiments, the fourth determining module 705 is specifically configured to:
[0203] According to at least one pruning node corresponding to each processing layer, pruning multiple initial processing nodes corresponding to each processing layer is performed to obtain a pruning model corresponding to the initial model, wherein the pruning model includes at least one target processing node corresponding to each processing layer;
[0204] Determine a decomposition model corresponding to the pruning model according to at least one target processing node corresponding to each processing layer;
[0205] By fine-tuning the model, the decomposition model is fine-tuned to obtain the target model.
[0206] In some possible embodiments, the fourth determining module 705 is specifically configured to:
[0207] For any processing layer, determine the target decomposition rank corresponding to each target processing node;
[0208] According to the target decomposition rank corresponding to each target processing node, the initial parameter matrix corresponding to each target processing node is decomposed to obtain multiple target parameter matrices corresponding to each target processing node;
[0209] The initial parameter matrix of each target processing node corresponding to each processing layer in the pruning model is updated to multiple target parameter matrices to obtain a decomposition model.
[0210] In some possible embodiments, the fourth determining module 705 is specifically configured to:
[0211] Get the preset compression rate;
[0212] According to the preset compression rate, determine the average rank corresponding to the pruned model;
[0213] The target decomposition rank is determined according to the average rank and the second importance corresponding to each target processing node.
[0214] The description of the features in the embodiment corresponding to the model compression device can refer to the relevant description of the embodiment corresponding to the model compression method, which will not be repeated here.
[0215] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the electronic device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus.
[0216] In the specific implementation process, at least one processor 801 executes the computer execution instructions stored in the memory 802, so that at least one processor 801 executes the above-mentioned model compression method embodiment.
[0217] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0218] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0219] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.
[0220] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.
[0221] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above-mentioned XX method embodiments when running.
[0222] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0223] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned model compression method embodiments are implemented.
[0224] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned model compression method embodiments are implemented.
[0225] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0226] The above is a detailed introduction to a model compression method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.
Claims
1. A model compression method, characterized in that: include: Acquire an initial model, wherein the initial model includes a plurality of processing layers, and each processing layer includes a plurality of initial processing nodes; Determining a first importance corresponding to each processing layer; Determine, according to the first importance corresponding to each processing layer, a node pruning rate corresponding to each processing layer, wherein the node pruning rate is a ratio of initial processing nodes reduced in the processing layer; Determining at least one pruning node corresponding to each processing layer according to the node pruning rate corresponding to each processing layer; According to at least one pruning node corresponding to each processing layer, a target model corresponding to the initial model is determined.
2. The method according to claim 1, characterized in that Determining a node pruning rate corresponding to each processing layer according to the first importance of each processing layer includes: Get the preset pruning rate; The node pruning rate corresponding to each processing layer is determined according to the preset pruning rate and the first importance corresponding to each processing layer.
3. The method according to claim 2, characterized in that Determining the node pruning rate corresponding to each processing layer according to the preset pruning rate and the first importance corresponding to each processing layer includes: Normalizing the first importance corresponding to each processing layer to obtain a normalized importance corresponding to each processing layer; Determining a pruning adjustment ratio corresponding to each processing layer according to the normalized importance corresponding to each processing layer; For any processing layer, the product of the pruning adjustment ratio corresponding to the processing layer and the preset pruning rate is determined as the node pruning rate corresponding to the processing layer.
4. The method according to any one of claims 1 to 3, characterized in that: For any processing layer, determining a first importance corresponding to the processing layer includes: determining a second importance of each initial processing node in the processing layer; The sum of the second importance of each initial processing node is determined as the first importance corresponding to the processing layer.
5. The method according to claim 1, characterized in that Determining at least one pruning node corresponding to each processing layer according to the node pruning rate corresponding to each processing layer includes: Determine, according to the node pruning rate corresponding to each processing layer, a plurality of intermediate pruning combinations corresponding to each processing layer, wherein the intermediate pruning combinations include at least one initial processing node; determining at least one processing layer set corresponding to the plurality of processing layers; At least one pruning node corresponding to each processing layer is determined according to the multiple intermediate pruning combinations corresponding to each processing layer and the at least one processing layer set.
6. The method according to claim 5, characterized in that For any processing layer; according to the node pruning rate corresponding to the processing layer, multiple intermediate pruning combinations corresponding to the processing layer are determined, including: Determining the number of pruned nodes of the pruned nodes according to the node pruning rate and the number of initial processing nodes corresponding to the processing layer; According to the number of pruned nodes, determine N initial pruning combinations from a plurality of initial processing nodes of the processing layer, each initial pruning combination includes M initial processing nodes, where M is the number of pruned nodes, N is an integer greater than or equal to 1, and M is an integer greater than or equal to 1; The N initial pruning combinations are optimized to obtain a plurality of intermediate pruning combinations corresponding to the processing layer.
7. The method according to claim 6, characterized in that Optimizing the N initial pruning combinations to obtain multiple intermediate pruning combinations corresponding to the processing layer, including: Determine a first loss difference corresponding to each of the N initial pruning combinations, where the first loss difference is used to indicate a difference between second output data after removing an initial processing node of the initial pruning combination in the initial model and second output data of the initial module; Determine the initial pruning combination whose first loss difference is less than the preset difference as the combination to be optimized; Iterative optimization processing is performed on the combination to be optimized to obtain multiple intermediate pruning combinations.
8. The method according to claim 5, characterized in that Determining at least one pruning node corresponding to each processing layer according to the multiple intermediate pruning combinations corresponding to each processing layer and the at least one processing layer set includes: Determining a target pruning combination corresponding to each processing layer according to the multiple intermediate pruning combinations corresponding to each processing layer and the at least one processing layer set; At least one pruning node corresponding to each processing layer is determined according to the target pruning combination corresponding to each processing layer.
9. The method according to claim 8, characterized in that The processing layer set includes T processing layers, where T is an integer greater than or equal to 1; the initial model has a total of L processing layers, and the number of the at least one processing layer set is , wherein L is an integer greater than or equal to 1; Wherein, for any processing layer set; according to a plurality of intermediate pruning combinations corresponding to each processing layer in the processing layer set, determining a target pruning combination corresponding to each processing layer includes: Determine a plurality of pruning sets corresponding to the processing layer set, wherein the pruning sets include T intermediate pruning combinations corresponding to the T processing layers; Determine a second loss difference corresponding to each pruning set; The T intermediate pruning combinations corresponding to the pruning set with the smallest second loss difference are determined as the target pruning combination corresponding to each processing layer in the processing layer set.
10. The method according to claim 1, characterized in that Determining a target model corresponding to the initial model according to at least one pruning node corresponding to each processing layer includes: According to the at least one pruning node corresponding to each processing layer, pruning the multiple initial processing nodes corresponding to each processing layer to obtain a pruning model corresponding to the initial model, wherein the pruning model includes at least one target processing node corresponding to each processing layer; Determining a decomposition model corresponding to the pruning model according to at least one target processing node corresponding to each processing layer; By fine-tuning the model, the decomposition model is fine-tuned to obtain the target model.
11. The method according to claim 10, characterized in that Determining a decomposition model corresponding to the pruning model according to at least one target processing node corresponding to each processing layer includes: For any processing layer, determine the target decomposition rank corresponding to each target processing node; Decomposing the initial parameter matrix corresponding to each target processing node according to the target decomposition rank corresponding to each target processing node to obtain multiple target parameter matrices corresponding to each target processing node; The initial parameter matrix of each target processing node corresponding to each processing layer in the pruning model is updated to the multiple target parameter matrices to obtain the decomposition model.
12. The method according to claim 11, characterized in that Determine the target decomposition rank corresponding to each target processing node, including: Get the preset compression rate; Determining an average rank corresponding to the pruning model according to the preset compression rate; The target decomposition rank is determined according to the average rank and the second importance corresponding to each target processing node.
13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the model compression method according to any one of claims 1 to 12 when executing the computer program.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the model compression method according to any one of claims 1 to 12.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the model compression method according to any one of claims 1 to 12 are implemented.