Model compression method, electronic equipment and computer readable storage medium
By performing mixed precision quantization (partial quantization and partial unquantization) on the target model, the problem of large storage consumption of network models is solved. This achieves the goal of maintaining model accuracy while reducing storage consumption under storage constraints, and is applicable to network models such as large language models.
Patent Information
- Application Number
- CN202410647915.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-23
- Publication Date
- 2025-11-25
AI Technical Summary
The large number of parameters, storage requirements, and computational demands of network models lead to high inference deployment costs, and existing technologies struggle to reduce storage requirements while maintaining model accuracy.
By quantizing and compressing the target model, the parameter matrices of some network layers are quantized, while the parameter matrices of the remaining network layers are not quantized, forming a target model with mixed precision quantization. The preset quantization algorithm and quantization error set are used for iterative optimization to ensure that storage usage is within limits and to reduce accuracy loss.
This approach ensures model accuracy while reducing storage footprint within storage constraints, solving the problem of large storage requirements in network models and making it suitable for large language models and other network models.
Smart Images
Figure CN121009931A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of model compression, in particular to a model compression method, an electronic device and a computer readable storage medium. BACKGROUND
[0002] With the development of technology, network models are used by more and more users. The model structure of the network model is also getting larger and larger. Based on this, the network model has the disadvantages of large parameter quantity, large storage occupation, large calculation quantity, high inference deployment cost and the like.
[0003] How to reduce the storage occupation of the network model is a problem that needs to be solved urgently. SUMMARY
[0004] The model compression method, the electronic device and the computer readable storage medium provided by the present application can not only guarantee the model accuracy, but also reduce the storage occupation of the model, and solve the problem of large storage occupation of the network model.
[0005] To solve the above technical problems, the present application provides a model compression method, which comprises: obtaining a trained target model; obtaining a storage limit corresponding to the target model; quantizing the parameter matrices of all network layers of the target model by using a preset quantization algorithm to obtain target quantization values of the parameter matrices corresponding to each network layer; obtaining quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain a quantization error set; obtaining a preset number of target quantization errors in the quantization error set, and modifying the target quantization values of the target network layers corresponding to the target quantization errors to default quantization values; wherein the default quantization values represent that the parameter matrices of the target network layers are not quantized; continuing to quantize the parameter matrices of the remaining network layers by using the preset quantization algorithm, again obtaining target quantization values of the parameter matrices corresponding to each remaining network layer, and performing the obtaining of the quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain the quantization error set, and iterating in this way until the quantized target model does not meet the storage limit.
[0006] By limiting the storage occupation of the quantized target model, in the process of quantizing and compressing the target model, the parameter matrices of part of the network layers of the target model are quantized and compressed, and the parameter matrices of the remaining part of the network layers are not quantized and compressed, and finally a hybrid precision quantized target model is formed, so as to reduce the precision loss caused by the compression of all parameter matrices on the target model, that is, the model accuracy can be guaranteed and the storage occupation of the model can be reduced under the storage limit, and the problem of large storage occupation of the network model is solved.
[0007] The parameter matrix includes a weight parameter matrix. The parameter matrix of all network layers of the target model is quantized by using a preset quantization algorithm to obtain a target quantization value of the parameter matrix corresponding to each network layer, including: the weight parameter matrix of all network layers of the target model is quantized by using a preset quantization algorithm to obtain a target quantization value of the weight parameter matrix corresponding to each network layer.
[0008] The quantization error of the weight parameter matrix corresponding to each network layer is obtained based on the target quantization value to obtain a quantization error set, including: an input feature vector corresponding to each network layer is obtained; a first value is obtained by using the target quantization value and the input feature vector; a second value is obtained by using the weight parameter matrix and the input feature vector; and the quantization error of the weight parameter matrix corresponding to each network layer is obtained based on the first value and the second value to obtain the quantization error set.
[0009] The first value is obtained by using the input feature vector and the target quantization value, and the second value is obtained by using the input feature vector and the weight parameter matrix, so as to obtain the quantization error caused by the quantization before and after.
[0010] The quantization error of the weight parameter matrix corresponding to each network layer is obtained based on the first value and the second value, including: a difference value between the first value and the second value is obtained; and a square of a 2-norm of the difference value is calculated to obtain the quantization error.
[0011] In response to the quantized target model not meeting the storage limit, the quantization value of the parameter matrix with the default quantization value in the current quantization is modified to the target quantization value, and the quantization operation is ended.
[0012] When the target model after the current quantization does not meet the storage limit, the storage occupation of the target model after the previous quantization belongs to the best storage occupation under the storage limit, so the quantization value after the previous quantization can be used as a criterion to end the quantization operation and obtain a target model with mixed precision quantization, thereby reducing the precision loss caused by the compression of all parameter matrices to the target model.
[0013] The default quantization value is 2, and the target quantization value is 1 / 2. The default quantization value represents that the elements in the parameter matrix of the corresponding network layer are stored in the form of 16 bits, and the target quantization value represents that the elements in the parameter matrix of the corresponding network layer are stored in the form of 4 bits.
[0014] The elements in the parameter matrix are stored in the form of 4 bits, and when reading, the reading response can be quickly performed.
[0015] The preset number of target quantization errors in the quantization error set are obtained, including: the preset number of target quantization errors are obtained from the quantization error set in descending order.
[0016] The parameter matrix causing large quantization error is selected from the quantization error set in descending order, and the parameter matrix is not quantized, so as to ensure the accuracy of the target model.
[0017] The target model is a large language model.
[0018] To solve the above technical problems, the present application provides an electronic device, which comprises a processor and a memory coupled to the processor; the memory is used to store a computer program, and the processor is used to execute the computer program to realize the method provided above.
[0019] To solve the above technical problems, the present application provides a computer readable storage medium, which stores a computer program, and the computer program realizes the method provided above when executed by a processor.
[0020] The model compression method, the electronic device and the computer readable storage medium provided by the present application limit the storage occupation of the quantized target model, so that in the process of quantizing and compressing the target model, the parameter matrix of part of the network layers of the target model is quantized and compressed, and the parameter matrix of the remaining part of the network layers is not quantized and compressed, and finally a hybrid precision quantized target model is formed, so as to reduce the accuracy loss caused by the compression of all parameter matrices on the target model, that is, the model precision can be ensured and the storage occupation of the model can be reduced under the storage limitation, and the problem of large network model storage occupation is solved. BRIEF DESCRIPTION OF DRAWINGS
[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0022] Figure 1 is a flowchart of an embodiment of the model compression method provided by the present application;
[0023] Figure 2 is a flowchart of another embodiment of the model compression method provided by the present application;
[0024] Figure 3 is a flowchart of an embodiment of the model compression method provided by the present application; Figure 2 is a flowchart of an embodiment of step 24 in the method provided by the present application;
[0025] Figure 4 is a structural diagram of an embodiment of the electronic device provided by the present application;
[0026] Figure 5Fig. 1 is a structural schematic diagram of an embodiment of the computer readable storage medium provided in the present application. DETAILED DESCRIPTION
[0027] The present application will be further described below in conjunction with the drawings and embodiments. It is particularly pointed out that the following embodiments are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following embodiments are only part of the embodiments of the present application, not all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of the present application.
[0028] In the description of the present application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly and specifically limited. The terms "first", "second", "third" in the embodiments of the present application are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first", "second", "third" can explicitly or implicitly include at least one of the features. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative positional relationship, movement condition, etc. between the components in a certain posture (as shown in the drawings), and if the certain posture changes, the directional indications also change accordingly. The terms "include" and "have" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or components inherent to the process, method, product or device.
[0029] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor does it necessarily refer to a particular embodiment in an exclusive sense. It is explicitly and implicitly understood by those skilled in the art that embodiments described herein can be combined with other embodiments.
[0030] With the development of technology, network models are used by more and more users. The model structure of the network model is also larger and larger. Based on this, the network model has the disadvantages of large parameter quantity, large storage occupation, large calculation quantity, and high inference deployment cost.
[0031] How to reduce the storage occupation of the network model is a problem that needs to be solved urgently.
[0032] The present inventors have found through long-term research that using a lower number of bits for all parameters of a model in the related art can cause a large precision loss, which can seriously reduce user experience and thus reduce product competitiveness.
[0033] Based on this, the present application proposes limiting the storage occupation of the quantized target model, so that in the quantization and compression process of the target model, the parameter matrices of part of the network layers of the target model are quantized and compressed, and the parameter matrices of the remaining part of the network layers are not quantized and compressed, and finally a hybrid precision quantized target model is formed, thereby reducing the precision loss caused by the compression of all parameter matrices on the target model, i.e., both the model precision and the storage occupation of the model can be ensured under the storage limit, solving the problem of large storage occupation of network models.
[0034] Referring to Figure 1 , Figure 1 is a flowchart of an embodiment of the model compression method provided by the present application. The method comprises:
[0035] Step 11: obtaining a trained target model.
[0036] In some embodiments, the target model can be a network model, such as a convolutional neural network model, a generative adversarial network model, and a series of network models trained by machine learning.
[0037] In some embodiments, the target model can be a large language model. A large language model (LLM, Large Language Model) is a deep learning model trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks such as text summarization, question answering, and translation.
[0038] Step 12: obtaining a storage limit corresponding to the target model.
[0039] In some embodiments, the storage limit can be set according to actual needs. For example, if the target model needs to be deployed in device A, the storage limit needs to be set according to the hardware specifications of device A. For example, if the target model needs to be deployed in device B, the storage limit needs to be set according to the hardware specifications of device B. It should be understood that the storage occupation corresponding to the storage limit is less than the current storage occupation corresponding to the target model. That is, the storage occupation corresponding to the target model before being quantized is greater than the storage limit.
[0040] Step 13: quantizing and compressing part of the network layers of the target model based on the storage limit.
[0041] In some embodiments, the partial network layer of the target model is quantized and compressed so that the storage occupation corresponding to the target model after quantization does not exceed the storage limit. That is, during the quantization process, when the partial network layer is quantized and the storage occupation corresponding to the target model is just less than the storage limit, the quantization of the target model is stopped.
[0042] In some embodiments, the quantization compression can be compressing the data corresponding to the partial network layer from the current bit to the target bit. For example, if the current bit is 16 bits, the target bit can be 1 bit, 2 bits, 4 bits or 8 bits. Specifically, the actual hardware specification and software specification of the device are determined.
[0043] In some embodiments, the parameter matrix and the activation of the partial network layer of the target model can be quantized.
[0044] In some embodiments, the parameter matrix of the partial network layer of the target model can be quantized.
[0045] In some embodiments, the activation of the partial network layer of the target model can be quantized.
[0046] In some embodiments, at least one of the AWQ (Activation-aware Weight Quantization, activation-aware weight quantization) method, the symmetric quantization method, and the asymmetric quantization method can be used to quantize the partial network layer of the target model.
[0047] In this embodiment, during the quantization and compression of the target model, the storage occupation after the quantization of the target model is limited. The partial network layer of the target model is quantized and compressed under the premise of the storage limit, and the remaining partial network layer is not quantized and compressed. Finally, the target model with mixed precision quantization is formed, so as to reduce the precision loss of the target model, that is, the storage occupation of the model can be reduced under the premise of ensuring the model precision, and the problem of large network model storage occupation is solved.
[0048] Referring to Figure 2 , Figure 2 is a flowchart of another embodiment of the model compression method provided by the present application. The method comprises:
[0049] Step 21: obtaining a trained target model.
[0050] Step 22: obtaining a storage limit corresponding to the target model.
[0051] Steps 21 to 22 have the same or similar technical solutions as any embodiment of the present application, and will not be repeated here.
[0052] Step 23: quantize the parameter matrices of all network layers of the target model by using a preset quantization algorithm to obtain target quantization values of the parameter matrices corresponding to each network layer.
[0053] In this embodiment, the parameter matrices of part of the network layers of the target model can be selected for quantization compression.
[0054] In some embodiments, the compressed parameter matrices can be quantized at the same compression ratio or at different compression ratios.
[0055] In some embodiments, when the target model is obtained, the parameter matrices of all network layers of the target model are first quantized by using a preset quantization algorithm, and then it is determined whether the current storage occupation of the target model after quantization of all network layers of the target model meets the storage limit. If yes, step 24 is performed. If no, the quantization of the target model is ended, the quantization method is improved, or the target model is retrained.
[0056] If the current storage occupation of the target model after quantization of all network layers of the target model meets the storage limit, it means that the compression of the target model is minimized and meets the storage limit, and then the precision problem can be considered. It is determined which parameter matrices can not be quantized within the storage limit to reduce the precision loss caused by quantization and ensure the precision of the target model. That is, it is determined which parameter matrices can not be quantized within the storage limit to make the storage occupation of the final target model infinitely close to or equal to the storage limit. In some embodiments, the storage occupation of the final target model can be equal to the capacity of the storage limit.
[0057] In the parameter matrices include weight parameter matrices, step 23 can be quantizing the weight parameter matrices of all network layers of the target model by using a preset quantization algorithm to obtain target quantization values of the weight parameter matrices corresponding to each network layer.
[0058] Step 24: obtain quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain a set of quantization errors.
[0059] After the parameter matrices are quantized, the precision of the target model is essentially damaged, and therefore it is necessary to determine the quantization errors of the parameter matrices before and after quantization.
[0060] In some embodiments, referring to Figure 3 , step 24 can be the following flow:
[0061] Step 31: obtain the input feature vectors corresponding to each network layer.
[0062] In some embodiments, the input feature vector corresponding to each network layer can be reserved during the training of the target model and directly used during quantization.
[0063] In some embodiments, the input feature vector corresponding to each network layer can be obtained by inputting the input in the training set into the quantized model after each training of the target model.
[0064] Step 32: obtaining a first value by using the target quantization value and the input feature vector.
[0065] In some embodiments, the first value can be obtained by the product of the target quantization value and the input feature vector. That is, the first value represents the data of the quantized model.
[0066] Step 33: obtaining a second value by using the weight parameter matrix and the input feature vector.
[0067] In some embodiments, the second value can be obtained by the product of the weight parameter matrix and the input feature vector. That is, the second value represents the data of the model before quantization.
[0068] Step 34: obtaining a quantization error corresponding to each network layer according to the first value and the second value, and obtaining a quantization error set.
[0069] In some embodiments, the difference between the first value and the second value can be obtained; the square of the 2-norm of the difference can be calculated to obtain the quantization error. That is, each network layer corresponds to a quantization error, and a quantization error set can be formed.
[0070] Step 25: obtaining a preset number of target quantization errors from the quantization error set, and modifying the target quantization value of the target network layer corresponding to the target quantization error to a default quantization value.
[0071] In some embodiments, the preset number of target quantization errors can be obtained from the quantization error set in descending order. By selecting the parameter matrices that cause large quantization errors from the quantization error set in descending order, these parameter matrices are not quantized, thereby ensuring the accuracy of the target model.
[0072] In some embodiments, the preset number of target quantization errors can be the quantization errors with the top ranking numbers in the quantization error set when the quantization errors are ranked in descending order of error. For example, if there are 100 quantization errors, the preset number can be 1, 2, 3, 5, or 10. Taking the preset number of 5 as an example, five largest quantization errors are selected from the 100 quantization errors as target quantization errors, and the target quantization value of the target network layer corresponding to the target quantization error is modified to a default quantization value.
[0073] In some embodiments, the preset number can be 1, the maximum quantization error in the set of quantization errors is selected as the target quantization error, and the target quantization value of the target network layer corresponding to the target quantization error is modified to the default quantization value.
[0074] The default quantization value represents that the parameter matrix of the target network layer is not quantized. Therefore, the parameter matrix with the default quantization value is not quantized in the final quantization.
[0075] Step 26: Continue to quantize the parameter matrices of the remaining network layers using the preset quantization algorithm, again obtain the target quantization value of the parameter matrix corresponding to each remaining network layer, and perform the quantization error of the parameter matrix corresponding to each network layer based on the target quantization value to obtain a set of quantization errors, and iterate until the quantized target model does not meet the storage limit.
[0076] In the iteration process, because the quantization value corresponding to the parameter matrix of part of the network layers is set to the default quantization value, i.e., the parameter matrix corresponding to the default quantization value is not quantized, the required storage occupancy of the target model increases from the minimum storage occupancy of quantization each time the iteration is performed, and when the required storage occupancy increases to exceed the storage limit, the iteration is ended. That is, in response to the quantized target model not meeting the storage limit, the quantization value of the parameter matrix with the default quantization value in the current iteration is modified to the target quantization value, and the quantization operation is ended. The parameter matrix with the default quantization value in the current iteration means the parameter matrix whose quantization value is modified from the target quantization value to the default quantization value after the current iteration. The parameter matrix whose quantization value is modified from the target quantization value to the default quantization value before does not belong to the parameter matrix with the default quantization value in the current iteration. For example, after the Nth iteration, the required storage occupancy of the target model exceeds the storage limit, the quantization value of the parameter matrix selected as the default quantization value in the Nth iteration is modified to the target quantization value. Finally, the parameter matrix with the target quantization value is quantized, and the quantization operation is ended.
[0077] That is, for example, after the Nth iteration, the required storage occupancy of the target model exceeds the storage limit, then in the N-1th iteration, the required storage occupancy of the target model is the closest and less than the storage limit. Therefore, it is restored to the case after the N-1th iteration. The parameter matrix with the target quantization value is quantized based on the quantization value after the N-1th iteration, and the parameter matrix with the default quantization value is not quantized, and the quantization operation is ended. That is, at this time, a set of quantization values can be obtained, and at this time, the weight parameter matrix of the target model can be quantized according to the set of quantization values.
[0078] In this embodiment, by limiting the storage occupation of the quantized target model, the parameter matrices of part of the network layers of the target model are quantized and compressed, and the parameter matrices of the remaining part of the network layers are not quantized and compressed, so that a hybrid precision quantized target model is finally formed, thereby reducing the precision loss caused by the compression of all parameter matrices of the target model, that is, the model precision can be ensured and the storage occupation of the model can be reduced under the storage limitation, thereby solving the problem of large storage occupation of the network model.
[0079] That is, by continuously iterating the quantization, the parameter matrices causing large quantization errors are selected and not quantized, so that the storage occupation of the target model formed by the quantized parameter matrices and the non-quantized parameter matrices can infinitely approach or equal the storage limitation, and the model precision loss is small.
[0080] In some embodiments, the default quantization value can be set to 2, and the target quantization value can be set to 1 / 2. The default quantization value represents that the elements in the parameter matrix of the corresponding network layer are stored in the form of 16 bits, and the target quantization value represents that the elements in the parameter matrix of the corresponding network layer are stored in the form of 4 bits. In some application scenarios, usually 1 byte corresponds to 8 bits, so the target quantization value represents that the elements in the parameter matrix of the corresponding network layer are stored in the form of 4 bits, and the reading can be quickly responded when reading.
[0081] In other embodiments, the default quantization value can be set based on the requirements of the storage hardware or software of the device where the target model is currently located. The target quantization value is set based on the requirements of the storage hardware or software of the device where the target model needs to be deployed. For example, the default quantization value is set to 2, and the target quantization value is set to 1; or the default quantization value is set to 2, and the target quantization value is set to 1 / 4; or the default quantization value is set to 2, and the target quantization value is set to 1 / 8.
[0082] In an application scenario, the model compression method of the present application is explained in combination with the following formula:
[0083] So that
[0084]
[0085]
[0086] First, a trained large language model is obtained, which is not limited to the type of large language model, such as QWen-7B, Llama-7B model, etc.
[0087] Then, given the storage limit B of the large language model, for example, B = 8 GiB, in order to reduce the quantization error as much as possible, the following optimization model is proposed:
[0088] So that
[0089]
[0090]
[0091] Where the operator represents the square of the 2-norm, and the operator |. | represents the number of elements of a vector or matrix; in formula 1, is a constraint condition, which is a limit on the model storage size, reflects the error caused by model quantization relative to the non-quantized model; the variable L represents the number of linear layers contained in the large language model, and the weight parameter matrix of the i-th linear layer is represented by W i ; the function Q(W i ) represents the quantization operation on the weight parameter matrix W i , and this does not limit the quantization method, for example, any third-party quantization method, such as the AWQ method, the symmetric quantization method, the asymmetric quantization method, etc. can be used to quantize the parameter matrix; x i represents the input feature vector of the i-th linear layer. α * represents the 2 L value range of α .
[0092] Then, according to the following algorithm, the above optimization model is solved by gradually finding local optimal solutions.
[0093]
[0094] Step 1, give a third-party quantization method, and quantize all the parameter matrices of the above large language model, at this time all the components of α are 1 / 2.
[0095] Step 2, judge whether the constraint condition is not satisfied, if the constraint condition is not satisfied, reassign the α i that was assigned to 2 in the last time to 1 / 2, terminate the iteration, and return α; if the constraint condition is satisfied, execute steps 3 and 4.
[0096] Step 3, select the N largest items in the L 2-norm squares in the objective function of formula 1, which correspond to α i of 1 / 2 (N ≥ 1), and set the corresponding α i to 2.
[0097] At this time, the objective function value will be reduced at most, that is, a is a local optimal solution under the current setting.
[0098] That is, the parameter matrix with large quantization error is selected, and it is not quantized, and on the basis of storage limitation, the corresponding reduction of the accuracy of the model is reduced.
[0099] Step 4, reusing the third-party quantization method used in step 1, removing a i The corresponding layer with a value of 2 is not actually quantized, and the other layers are re-quantized. Step 2 is continued.
[0100] That is, steps 1 to 4 above essentially combine the greedy algorithm, find several weight parameter matrices with large quantization errors in each iteration process, and filter them out, and do not quantize them in the final quantization, so as to reduce the accuracy loss caused by the quantization of the target model.
[0101] Combining the above formula is: in the first iteration, all the parameter matrices of the large language model are quantized, and at this time, all the components of a are 1 / 2. Then, combining formula 1, determine the maximum N corresponding to a i 1 / 2 term in the square of the two norms. Set the corresponding a i to 2. That is, these parameter matrices with large quantization errors will not be quantized. Then, combining formula 1 and formula 2, because a i is set to 2, when formula 1 is used, the value of a is determined by formula 2. It can be seen that a i is set to 2, is equal to W i , in formula 1, the error is 0, which is equivalent to the corresponding parameter matrix not being quantized. Repeat the iteration, find the parameter matrix that affects the model accuracy under the storage limit, do not quantize it, and quantize other parameter matrices, then a model with high precision and small storage capacity can be obtained.
[0102] In this embodiment, during the quantization compression of the target model, the storage occupation of the quantized target model is limited, and the partial network layer of the target model is quantized under the premise of storage limitation, and the remaining network layer is not quantized. Compression, finally forming a hybrid precision quantization target model, so as to reduce the accuracy loss of the target model, that is, the storage occupation of the model can be reduced under the premise of ensuring the accuracy of the model, solving the problem of large network model storage occupation.
[0103] Further, the optimal mixed precision target model is found by establishing a mathematical model (formula 1 described above), which is more reliable than selecting certain layers with higher precision based on experience. Because the number of layers of a large language model is several hundred, it is impossible to manually select all combinations to obtain the optimal combination.
[0104] Further, the model provided in the present application has an effective range, and the storage limit can be arbitrarily specified. The larger the general limit storage size, the higher the precision of the quantized model. This facilitates obtaining a quantized model that occupies relatively less storage under the condition of a preset precision.
[0105] Further, the greedy algorithm can obtain a local optimal solution in a short time, making it possible to solve the problem by exhaustion.
[0106] Further, finding a quantized model with relatively good precision according to the storage limit is conducive to deploying the model on hardware with lower cost, and is also conducive to improving throughput on hardware with larger storage.
[0107] Referring to Figure 4 , Figure 4 is a structural schematic diagram of an embodiment of an electronic device provided in the present application. The electronic device 40 includes a processor 41 and a memory 42 coupled to the processor 41; wherein the memory 42 is used to store a computer program, and the processor 41 is used to execute the computer program to implement the following method:
[0108] obtaining a trained target model; obtaining a storage limit corresponding to the target model; quantizing parameter matrices of all network layers of the target model using a preset quantization algorithm to obtain target quantization values of the parameter matrices corresponding to each network layer; obtaining quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain a quantization error set; obtaining a preset number of target quantization errors in the quantization error set, and modifying target quantization values of target network layers corresponding to the target quantization errors to default quantization values; wherein the default quantization value represents that the parameter matrices of the target network layers are not quantized; continuing to quantize parameter matrices of the remaining network layers using the preset quantization algorithm, again obtaining target quantization values of the parameter matrices corresponding to each remaining network layer, and performing obtaining quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain a quantization error set, and iterating in this way until the quantized target model does not meet the storage limit.
[0109] It can be understood that the processor 41 is used to execute the computer program and can also implement the method of any of the above embodiments.
[0110] Referring to Figure 5 , Figure 5is a structural schematic diagram of an embodiment of the computer readable storage medium provided in the present application. The computer readable storage medium 50 stores a computer program 51, and the computer program 51 realizes the following method when executed by a processor:
[0111] obtaining a trained target model; obtaining a storage limit corresponding to the target model; quantizing parameter matrices of all network layers of the target model by using a preset quantization algorithm to obtain target quantization values of the parameter matrices corresponding to each network layer; obtaining quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain a quantization error set; obtaining a preset number of target quantization errors in the quantization error set, and modifying target quantization values of a target network layer corresponding to the target quantization errors to default quantization values; wherein the default quantization values represent that the parameter matrices of the target network layer are not quantized; continuing to quantize the parameter matrices of the remaining network layers by using the preset quantization algorithm, again obtaining target quantization values of the parameter matrices corresponding to each remaining network layer, and performing obtaining quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain a quantization error set, and iterating in this way until the quantized target model does not meet the storage limit.
[0112] It can be understood that the computer program 51, when executed by the processor, can also realize the method of any of the above embodiments.
[0113] In summary, the model compression method, the electronic device, and the computer readable storage medium provided in the present application limit the storage occupancy of the quantized target model, so that in the process of quantizing and compressing the target model, the parameter matrices of part of the network layers of the target model are quantized and compressed, and the parameter matrices of the remaining part of the network layers are not quantized and compressed, and finally a hybrid precision quantized target model is formed, thereby reducing the precision loss caused by compressing all parameter matrices of the target model, that is, the model precision can be guaranteed and the storage occupancy of the model can be reduced under the storage limit, thereby solving the problem of large storage occupancy of the network model.
[0114] Further, the optimal hybrid precision target model is found by establishing a mathematical model (formula 1, formula 2, and formula 3 described above), which is more reliable than selecting certain layers with higher precision based on experience. Because the number of layers of a large language model is several hundred, it is impossible to exhaustively select all combinations to obtain the optimal combination.
[0115] Further, the model provided in the present application can be arbitrarily specified within an effective range. Generally, the larger the storage size limit, the higher the precision of the quantized model. This facilitates obtaining a quantized model with relatively small storage occupancy under a preset precision condition.
[0116] Further, the greedy algorithm can obtain a local optimal solution in a relatively short time, making it possible to solve the problem exhaustively.
[0117] Further, finding a quantization model with relatively good accuracy according to the storage limit is conducive to deploying the model on hardware with lower cost, and is also conducive to improving throughput on hardware with larger storage.
[0118] In several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other manners. For example, the above-described device embodiments are merely illustrative, and the division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0119] The integrated units in the above other embodiments, if realized in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, and various other media that can store program codes.
[0120] The above only describes some embodiments of the present application, and does not limit the protection scope of the present application, and any equivalent device or equivalent flow transformation using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A model compression method, characterized in that, The method comprises: acquiring a trained target model; acquiring a storage limit corresponding to the target model; quantizing parameter matrices of all network layers of the target model using a preset quantization algorithm to obtain target quantization values of the parameter matrices corresponding to each network layer; based on the target quantization values, acquiring quantization errors of the parameter matrices corresponding to each network layer to obtain a quantization error set; acquiring a preset number of target quantization errors in the quantization error set, and modifying target quantization values of a target network layer corresponding to the target quantization errors to default quantization values; wherein the default quantization values represent that the parameter matrices of the target network layer are not quantized; continuing to quantize parameter matrices of remaining network layers using the preset quantization algorithm, again obtaining target quantization values of the parameter matrices corresponding to each remaining network layer, and performing the acquisition of quantization errors of the parameter matrices corresponding to each network layer based on the target quantization values to obtain a quantization error set, and iterating until the quantized target model does not meet the storage limit.
2. The method of claim 1, wherein, The parameter matrices comprise weight parameter matrices, and the quantizing the parameter matrices of all network layers of the target model using the preset quantization algorithm to obtain the target quantization values of the parameter matrices corresponding to each network layer comprises: quantizing weight parameter matrices of all network layers of the target model using the preset quantization algorithm to obtain target quantization values of the weight parameter matrices corresponding to each network layer.
3. The method of claim 2, wherein, The acquisition of the quantization errors of the weight parameter matrices corresponding to each network layer based on the target quantization values comprises: acquiring an input feature vector corresponding to each network layer; obtaining a first value using the target quantization values and the input feature vector; obtaining a second value using the weight parameter matrices and the input feature vector; acquiring a quantization error of the weight parameter matrices corresponding to each network layer according to the first value and the second value to obtain a quantization error set.
4. The method of claim 3, wherein, The acquisition of the quantization error of the weight parameter matrices corresponding to each network layer according to the first value and the second value comprises: acquiring a difference value between the first value and the second value; calculating a square of a 2-norm of the difference value to obtain a quantization error.
5. The method of claim 1, wherein, In response to the quantized target model not meeting the storage limit, the quantization values of the parameter matrices that are the default quantization values in the current iteration are modified back to the target quantization values, and the quantization operation is ended.
6. The method of claim 1, wherein, The default quantization value is 2, and the target quantization value is 1 / 2. The default quantization value represents that elements in the parameter matrices of the corresponding network layer are stored in the form of 16 bits, and the target quantization value represents that elements in the parameter matrices of the corresponding network layer are stored in the form of 4 bits.
7. The method of claim 1, wherein, The acquisition of the preset number of target quantization errors in the quantization error set comprises: acquiring the preset number of target quantization errors from the quantization error set in descending order.
8. The method according to any one of claims 1 to 7, characterized in that, The target model is a large language model.
9. An electronic device, comprising: The electronic device comprises a processor and a memory coupled to the processor; wherein the memory is configured to store a computer program, and the processor is configured to execute the computer program to implement the method according to any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the method according to any one of claims 1-8.