Model weight reduction method and system for end-side large model

Abstract

The invention relates to the technical field of large models, in particular to a model lightweight method and system for an end-side large model, and aims to solve the problem of how to obtain adaptive quantization bits based on the precision requirement of each layer of the end-side large model. In order to achieve the purpose, the method comprises the steps that reference precision and a data distribution statistical result corresponding to a preset quantization bit are obtained based on a reasoning sample set, disturbance data, the reference precision and the data distribution statistical result are obtained based on a reasoning data set, the disturbance data, the reference precision and the data distribution statistical result, and then the optimal bit of each layer is obtained. The large model is quantized by using the optimal bit adaptive to the precision of each layer, and the quantized bit can be reduced for the layer with low requirement on operator precision, so that the operation speed and the storage efficiency are improved; and for the layer with high operator precision requirement, the quantization bit can be improved, and the precision loss caused by fixed quantization with lower numerical value is made up, so that the operation effect of the end side large model is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of large models, and in particular to a method and system for lightweighting a large end-side model. Background Art

[0002] Deep learning has been widely applied across various branches of AI technology, providing sufficient support for industrial-grade productization. The rise of "big models" in AI has taken the field a step further on the path to intelligentization. However, while big models improve performance, they also require significantly more storage and computational resources, increasing their dependence on hardware platforms. Currently, mainstream big model technologies in the industry are mostly based on large-scale servers, allowing users to time-share and reuse the server's extensive computing and storage resources. However, this also introduces network dependencies and increased costs.

[0003] To enable the deployment of large models on edge devices, reduce their reliance on networks and hardware platforms, and facilitate user sharing of the results of large model deployments, companies such as NVIDIA and Intel have introduced inference and quantization technologies specifically designed for large models deployed on edge devices. However, most quantization technologies are limited to global quantization, using the same number of bits for quantization at each layer and failing to meet the accuracy requirements of each layer. Therefore, how to quantize with a bit count appropriate to the accuracy requirements of each layer in large edge models, thereby simultaneously meeting the requirements for accuracy and runtime and storage efficiency, has become a pressing issue.

[0004] Accordingly, this field requires a new model lightweighting solution for large end-side models to solve the above problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problem of how to obtain quantization bits that are compatible with the accuracy requirements of each layer based on the accuracy requirements of each layer of the end-side large model.

[0006] In a first aspect, a method for lightweighting a large end-side model is provided, the method comprising: quantizing the first model based on the second bit to obtain a reference target model, wherein the first model is constructed based on the first bit and has been trained based on the first sample set; Performing inference on the first model based on the first inference sample set to obtain statistical results of layer data distribution of each layer; Inferring the first model and the reference target model based on the second inference sample set, and combining the K disturbance data corresponding to the K error level degradation coefficients of each layer and the statistical results of the layer data distribution to obtain K disturbance influence factors corresponding to the K error level degradation coefficients of each layer; Determining the optimal bit position of each layer based on the K disturbance impact factors of each layer; quantizing the first model based on the optimal bit positions of each layer to obtain a target model; The first reasoning sample set and the second reasoning sample set are respectively constructed based on the first sample set.

[0007] In one technical solution of the above-mentioned method for lightweighting the large end-side model, the method for obtaining the disturbance impact factor includes: Performing inference on the first model and the reference target model based on the second inference sample set to obtain a layer reference accuracy loss of each layer and a reference cumulative accuracy loss of a final layer; Based on the second inference sample set and in combination with the K perturbation data corresponding to the K error level degradation coefficients of each layer, reasoning on the first model and the reference target model to obtain the layer perturbation accuracy loss corresponding to each error level degradation coefficient of each layer and the perturbation cumulative accuracy loss of the final layer; Based on the layer disturbance accuracy loss and the disturbance cumulative accuracy loss corresponding to each error level degradation coefficient of each layer, and combined with the layer reference accuracy loss, the reference cumulative accuracy loss and the layer data distribution statistical results, the disturbance impact factor corresponding to each error level degradation coefficient of each layer is obtained.

[0008] In one technical solution of the above-mentioned method for lightweighting a large model on the end side, the method for obtaining the layer reference accuracy loss includes: Obtain the output data l of the i-th layer of the reference target model corresponding to each sample in the second reasoning sample set i,n , get the first output tensor data L i ; Get the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set Will As the input data of the i-th layer of the reference target model, obtain the output data of the i-th layer of the reference target model corresponding to each sample in the second reasoning sample set Get the second output tensor data in, OOP i is the operator of the i-th layer of the reference target model; Based on the first output tensor data L i and the second output tensor data The similarity between them is used to obtain the reference accuracy loss of the i-th layer; Wherein, both the first model and the reference target model have an I-layer structure, 1≤i≤I; The second inference sample set includes N samples, 0≤n≤N-1, and i and n are both integers.

[0009] In one technical solution of the model lightweighting method for the aforementioned large-scale end-side model, the first model and the reference target model both have a one-layer structure, the final layer is the first layer, and a method for obtaining the reference cumulative precision loss of the final layer includes: Obtain the output data l of the first layer of the reference target model corresponding to each sample in the second reasoning sample set I,n , get the first final output tensor data L I ; Obtain the output data of the first layer of the first model corresponding to each sample in the second inference sample set Get the second final output tensor data Based on the first final output tensor data L I and the second final output tensor data The similarity between them is used to obtain the reference cumulative precision loss of the final layer; The second inference sample set includes N samples, 0≤n≤N-1, and n is an integer.

[0010] In one technical solution of the above-mentioned method for lightweighting a large model on the end side, the method for obtaining the layer perturbation accuracy loss includes: Obtain the output data l of the i-th layer of the reference target model corresponding to each sample in the second reasoning sample set i,n , get the first output tensor data L i ; Get the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set based on and the perturbation data E corresponding to the error level degradation coefficient k of the i-th layer i-1,k , construct the input data of the i-th layer of the reference target model Will As the input data of the i-th layer of the reference target model, obtain the output data l′ of the i-th layer of the reference target model corresponding to the error level degradation coefficient k i,n (E i-1,k ), get the third output tensor data L' i (E i-1,k ),in, OOP iis the operator of the i-th layer of the reference target model; Based on the first output tensor data L i and the third output tensor data L′ i (E i-1,k ), and obtain the layer disturbance accuracy loss corresponding to the error level degradation coefficient k of the i-th layer; Wherein, both the first model and the reference target model have an I-layer structure, 1≤i≤I; The second inference sample set includes N samples, 0≤n≤N-1; The number of the error level degradation coefficients is K, 0≤k≤K-1, and i, n and k are all integers.

[0011] In one technical solution of the model lightweighting method for the aforementioned large-scale end-side model, the first model and the reference target model both have a one-layer structure, the final layer is the first layer, and a method for obtaining the cumulative perturbation precision loss of the final layer includes: Obtain the output data l of the first layer of the reference target model corresponding to each sample in the second reasoning sample set I,n , get the first final output tensor data L I ; Get the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set based on and the perturbation data E corresponding to the error level degradation coefficient k of the i-th layer i-1,l , construct the input data of the i-th layer of the first model Will As the input data of the i-th layer of the first model, obtain the output data of the i-th layer of the first model Get the third final output tensor data Based on the first final output tensor data L I and the third final output tensor data The similarity between them is used to obtain the disturbance cumulative accuracy loss of the final layer corresponding to the error level degradation coefficient k of the i-th layer; The second inference sample set includes N samples, 0≤n≤N-1; The number of the error level degradation coefficients is K, 0≤k≤K-1; 1≤i≤I, i, n and k are all integers.

[0012] In one technical solution of the above-mentioned method for lightweighting the large end-side model, the method further includes: The disturbance data is constructed based on a preset error initial value and a negative integer exponential power, wherein the negative integer exponential power is constructed based on the error level degradation coefficient k, 0≤k≤K-1, and k is an integer.

[0013] In one technical solution of the above-mentioned method for lightweighting a large model on the device side, the first sample set includes a first training sample set, a first test sample set, and a first validation sample set. The method further includes: Constructing the first inference sample set based on the first training sample set, the first test sample set, extreme samples, and samples generated by an AI tool; Constructing the second reasoning sample set based on the first verification sample set; The samples in the first reasoning sample set and the samples in the second reasoning sample set are different, and the number of samples in the first reasoning sample set is greater than the number of samples in the second reasoning sample set.

[0014] In one technical solution of the aforementioned method for lightweighting a large end-side model, "determining the optimal bit position for each layer based on the K disturbance impact factors for each layer" includes: Comparing the K disturbance impact factors of each layer respectively; The value of the error level degradation coefficient k corresponding to the disturbance influence factor with the smallest value is the optimal bit position of the layer, where 0≤k≤K-1, and k is an integer.

[0015] In a second aspect, a large-scale model system is provided, comprising: A quantitative reasoning library, comprising: operators of each layer in the first model, operators of each layer in the reference target model, and operators of each layer in the target model; a quantization optimization module, the quantization optimization module being configured to obtain the target model based on the quantization inference library by using the model lightweight quantization method for the large end-side model described in any of the above technical solutions; An inference execution module is configured to control the execution and scheduling of the target model based on the quantized inference library.

[0016] One or more of the above-mentioned technical solutions of the present application have at least one or more of the following beneficial effects: based on the accuracy requirements of each layer of the end-side large model, the quantization bits of each layer can be set to different values. For layers with low operator accuracy requirements, the quantization bits can be appropriately reduced, thereby improving the computing speed and storage efficiency, and improving the operating efficiency of the large model; for layers with high operator accuracy requirements, the quantization bits can be appropriately increased, which to a certain extent compensates for the accuracy loss caused by using a lower-value, fixed quantization, thereby ensuring the operating effect of the end-side large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The disclosure of this application will become more easily understood with reference to the accompanying drawings. Those skilled in the art will readily understand that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0018] Figure 1 This is a flow chart of the main steps of a method for lightweighting a large end-side model according to an embodiment of the present application.

[0019] Figure 2 It is a detailed flowchart of step S103 according to an embodiment of the present application.

[0020] Figure 3 It is a flowchart of the detailed steps of a method for obtaining layer reference accuracy loss according to an embodiment of the present application.

[0021] Figure 4 It is a flowchart of detailed steps of a method for obtaining layer perturbation accuracy loss according to an embodiment of the present application.

[0022] Figure 5 It is a structural diagram of an end-side large model system according to another embodiment of the present application. DETAILED DESCRIPTION

[0023] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the scope of protection of the present application.

[0024] In the description of this application, "module" and "processor" may include hardware, software, or a combination of both. A module may include hardware circuitry, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. A processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor may be implemented in software, hardware, or a combination of both. Computer-readable storage media include any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" refers to all possible combinations of A and B, such as only A, only B, or both A and B. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or both A and B. The singular forms "a" and "the" may also include the plural forms.

[0025] See attached Figure 1 , Figure 1 This is a flow chart of the main steps of a method for lightweighting a large end-side model according to an embodiment of the present application. Figure 1 As shown, the method for lightweighting the large model on the end side in the embodiment of the present application includes: Step S101: quantizing the first model based on the second bit to obtain a reference target model; Step S102: performing inference on the first model based on the first inference sample set to obtain statistical results of layer data distribution of each layer; Step S103: Inferring the first model and the reference target model based on the second inference sample set, and combining the K disturbance data corresponding to the K error level degradation coefficients of each layer and the layer data distribution statistics to obtain the K disturbance influence factors corresponding to the K error level degradation coefficients of each layer; Step S104: determining the optimal bit position of each layer based on the K disturbance impact factors of each layer; Step S105: quantize the first model based on the best bit positions of each layer to obtain a target model.

[0026] In step S101, first, it is necessary to select the large model (first model) that needs to be deployed on the end-side device based on the scenario and business requirements. This application does not limit the type of the first model. The first model can be a mature public large model or a large model developed by the user. As an example, the first model can be a language large model, such as the DeepSeek model, the ChatGPT model, etc.; it can also be a visual large model, such as the CLIP model, the SAM model, etc., or a multimodal large model, such as the Llama model, the Gemini model, etc.

[0027] The first model is constructed based on the first bit position and has been trained using the first sample set, i.e., the parameters of each layer of the first model have been determined. The first bit position is typically relatively high, such as 32 bits corresponding to Float32 or 64 bits corresponding to Float64, so that the first model, deployed on a cloud server with abundant storage and computing resources, can provide users with high-precision computing services.

[0028] For end-side devices, such as tablets, smartphones, and edge computing devices, the first model often cannot be directly deployed on the device due to device hardware resource limitations. In this case, the first model can be inferred and quantized based on a lower bit (the second bit), thereby reducing the storage and computing required for the deployment and independent use of the target model (the end-side large model obtained by quantizing the first model with low bits), thereby reducing the requirements for device hardware resources.

[0029] In an embodiment of the present application, the first model can be quantized based on the 8 bits (second bit) corresponding to int8 to obtain a reference target model. At this time, the parameters of each layer in the reference target model are quantized based on the same bit (int8).

[0030] Next, in combination with steps S102 to S105, it will be explained how, based on the reference target model, the method for obtaining the actual quantization bits required for each layer can be further optimized according to the accuracy required by each layer, so that the final target model can not only meet the accuracy requirements required by the user, but also meet the resource requirements of reducing storage and computing power.

[0031] Before executing step S102 (or before executing step S101), it is necessary to construct an inference sample set for acquiring the target model inference. Considering that the first model has been trained based on the first sample set, the inference sample set is constructed based on the first sample set.

[0032] Machine learning (large model training) sample sets are typically divided into training sample sets (training sets), test sample sets (test sets), and validation sample sets (validation sets) based on their functions. Training sample sets and test sample sets are typically used to train and evaluate trained models, respectively, while validation sample sets are typically used to verify model performance and optimize model parameters. The first sample set also includes the first training sample set, the first test sample set, and the first validation sample set.

[0033] With reference to the division of the first sample set, the inference sample set is divided into a first inference sample set and a second inference sample set. The first inference sample set is constructed based on the first training sample set and the first test sample set, and is mainly used to evaluate the data distribution of the first model; the second inference sample set is constructed based on the first verification sample set, and is mainly used to evaluate the impact of perturbation data on the accuracy of the first model and the quantized first model, and to optimize and adjust the bits required for the final quantization of each layer; and the samples in the first inference sample set and the samples in the second inference sample set are different.

[0034] Based on the size of the first model, select a first inference sample set and a second inference sample set with appropriate sample sizes, and ensure that the number of samples in the first inference sample set is greater than the number of samples in the second inference sample set. In this embodiment of the present application, the number of samples M in the first inference sample set is 200, and the number of samples N in the second inference sample set is 20.

[0035] It should be noted that the number of samples M in the first inference sample set and the number of samples N in the second inference sample set can also be selected as other values, such as M=300, N=25, etc. Generally, the larger the scale of the first model, the larger the values ​​of M and N.

[0036] In addition, a small number of extreme samples and at least one of samples generated by AI tools (AI samples) can be added to the first inference sample set to help the target model build and refine the classification boundaries and improve its generalization ability.

[0037] As an example, 2% of extreme samples (e.g., completely black or completely white images) and 5% of AI samples can be added to the first inference sample set. Depending on actual circumstances, those skilled in the art may also choose other proportions of extreme samples and AI samples, such as 3% of extreme samples and / or 6% of AI samples. Without departing from the principles of this application, these modified or replaced technical solutions will fall within the scope of protection of this application.

[0038] In step S102, the first model is inferred based on the M samples in the first inference sample set, and the output data of each sample corresponding to each layer in the inference process is recorded.

[0039] The numerical distribution of the output data of all samples corresponding to each layer of the first model is statistically analyzed to obtain the layer data distribution statistical results of each layer. The layer data distribution statistical results specifically include: layer maximum value VMAX i , layer minimum value VMIN i and layer standard deviation δ i , where i is the number of layers of the first model, 1≤i≤I, and the first model has an I-layer structure.

[0040] It should be noted that in the embodiment of the present application, the first model has a 1-layer structure, and the reference target model and the target model also have a 1-layer structure, and the layers of the first model, the reference target model and the target model correspond one to one. The operators of each layer of the first model are denoted as OP i In step 101, the operators of each layer of the reference target model quantized based on the second bit (int8) are recorded as OOP i .

[0041] Continue reading Figure 2 , combined with Figure 2 The specific method of step S103 is described. In step S1031, the first model and the reference target model are inferred using the second inference sample set to obtain the layer reference precision loss LP of each layer. i and the reference cumulative accuracy loss AP of the final layer I .

[0042] Next, combine Figure 3 Explain the layer reference precision loss LP of each layer iIn step S301, the output data l of the i-th layer of the reference target model corresponding to each sample in the second inference sample set (the N samples in the second inference sample set need to be quantized based on int8 before being input into the reference target model) is obtained. i,n , get the first output tensor data L corresponding to all samples in the second inference sample set i ; L i =[l i,0 l i,1 … l i,n … l i,N-1 ] (Formula 1) Among them, l i,n Represents the output data of the i-th layer of the reference target model corresponding to the n-th sample, and there is l i,n =OOP i (l i-1,n ) relationship, 1≤i≤I, 0≤n≤N-1.

[0043] In step S302, first, the N samples in the second inference sample set are inferred during the inference process of the first model, and the output tensor data corresponding to each layer are calculated. in, Represents the output tensor data of the i-th layer of the first model corresponding to the n-th sample, there is The relationship is, 1≤i≤I, 0≤n≤N-1.

[0044] According to Formula 2, the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set can be obtained

[0045] In step S303, As the input data of the i-th layer of the reference target model, obtain the output data of the i-th layer of the reference target model corresponding to each sample in the second inference sample set Get the second output tensor data corresponding to all samples in the second inference sample set in,

[0046] In step S304, based on the first output tensor data L i and the second output tensor data The similarity between them is used to obtain the layer reference precision loss LP of the i-th layer. i , in, Indicates Li and The cosine similarity between them, 1≤i≤I.

[0047] Since both the first model and the reference target model are 1-layer structures, the final layer is the 1st layer. According to the aforementioned step S301, the output data of the 1st layer of the reference target model corresponding to each sample in the second inference sample set is obtained. I,n , get the first final output tensor data L corresponding to all samples in the second inference sample set I , L I =[l I,0 l I,1 … l I,n … l I,N-1 ] (Formula 5).

[0048] According to the aforementioned step S302, the output data of the first layer of the first model corresponding to each sample in the second inference sample set is obtained. Get the second final output tensor data corresponding to all samples in the second inference sample set

[0049] Based on the first final output tensor data L I and the second final output tensor data The similarity between them is used to obtain the reference cumulative accuracy loss AP of the final layer (layer I) I , Among them, 1≤i≤I.

[0050] Similarly, the reference cumulative accuracy loss AP of the i-th layer can be defined based on equations 1 and 2. i , Where 1≤i≤I. When i=I, the reference cumulative accuracy loss AP of the final layer (layer I) can be obtained I .

[0051] In step S1032, the first model and the reference target model are inferred using the second inference sample set and combining the K disturbance data corresponding to the K error level degradation coefficients of each layer to obtain the layer disturbance accuracy loss corresponding to each error level degradation coefficient of each layer and the disturbance cumulative accuracy loss of the final layer.

[0052] In order to evaluate the impact of disturbance data of different numerical values ​​on the first model and the reference target model, K sequentially degraded disturbance data can be constructed for each layer based on the preset initial error value and the error level degradation coefficient. The kth disturbance data of the i-th layer can be expressed as: E i-1,k =random(0,R*2 -k ) (Formula 9) Among them, random(0,R*2 -k ) means to generate a value in (0,R*2 -k ), i is the layer value, k is the error level degradation coefficient, and R is the preset error initial value.

[0053] In the embodiment of the present application, the number of layers of the first model and the reference target model is I, that is, the value range of i is 1≤i≤1. Considering that the reference target model is quantized based on the second bit (int8), the value of K can be a value greater than the second bit. As an example, K=10, that is, the value range of k is 0≤k≤10, where k is an integer.

[0054] In addition, considering that the sample data of this application need to be normalized before being input into the model (including the first model and the reference target model), R can take a value within the range of (0, 1). As an example, R = 0.8. Those skilled in the art can also select other values ​​of R according to actual conditions, for example, R = 0.6 or R = 0.4.

[0055] Specifically, when R=0.8, the K(10) disturbance data for the i-th layer include: a random number in the interval (0, 0.8) corresponding to the error level degradation coefficient k=0, a random number in the interval (0, 0.4) corresponding to the error level degradation coefficient k=1, a random number in the interval (0, 0.2) corresponding to the error level degradation coefficient k=2, a random number in the interval (0, 0.1) corresponding to the error level degradation coefficient k=3, a random number in the interval (0, 0.05) corresponding to the error level degradation coefficient k=4, and so on.

[0056] Next, combine Figure 4 The method for obtaining the layer perturbation accuracy loss corresponding to the error level degradation coefficient k of the i-th layer is described. In step S401, the output data l of the i-th layer of the reference target model corresponding to each sample in the second inference sample set (the N samples in the second inference sample set need to be quantized based on int8 before being input into the reference target model) is obtained. i,n , get the first output tensor data L i ; L i =[l i,0 l i,1… l i,n … l i,N-1 ] (Equation 10) Among them, l i,n Represents the output data of the i-th layer of the reference target model corresponding to the n-th sample, and there is l i,n =OOP i (l i-1,n ) relationship, 1≤i≤I, 0≤n≤N-1.

[0057] In step S402, first, the N samples in the second inference sample set are inferred during the inference process of the first model, and the output tensor data corresponding to each layer are calculated. in, Represents the output tensor data of the i-th layer of the first model corresponding to the n-th sample, there is The relationship is, 1≤i≤I, 0≤n≤N-1.

[0058] According to formula 11, the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set can be obtained

[0059] In step S403, based on and the perturbation data E corresponding to the error level degradation coefficient k of the i-th layer i-1,k , construct the input data of the i-th layer of the reference target model

[0060] In step S404, As the input data of the i-th layer of the reference target model, obtain the output data l′ of the i-th layer of the reference target model corresponding to the error level degradation coefficient k i,n (E i-1,k ), obtain the third output tensor data L′ of all samples in the second inference sample set corresponding to the error level degradation coefficient k of the i-th layer i (E i-1,k ), in,

[0061] In step S405, based on the first output tensor data L i And the third output tensor data L′ i (E i-1,k ), and obtain the layer perturbation accuracy loss LP corresponding to the error level degradation coefficient k of the i-th layer i,k , Among them, 1≤i≤I, 0≤k≤K-1.

[0062] According to the aforementioned step S401, the output data l of the first layer of the reference target model corresponding to each sample in the second reasoning sample set is obtained. I,n , get the first final output tensor data L I , L I =[l I,0 l I,1 … l I,n … l I,N-1 ] (Formula 14).

[0063] According to the above step S402 and formula 11, the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set is obtained. Combined with the perturbation data E corresponding to the error level degradation coefficient k of the i-th layer i-1,k , construct the input data of the i-th layer of the first model

[0064] Will As the input data of the i-th layer of the first model, get the output data of the final layer (layer I) Get the third final output tensor data of all samples in the second inference sample set corresponding to the error level degradation coefficient k of the i-th layer

[0065] Specifically, each layer before the i-1th layer (including the i-1th layer) uses each sample in the second inference sample set to infer the first model; each layer after the i-th layer (including the i-th layer) uses the perturbation data E directly or indirectly. i-1,k The aforementioned indirection includes the result of perturbed data E i-1,k Influence.

[0066] At this time, the output tensor data of the i-1th layer of the first model is Add perturbation data E i-1，k Input tensor data of the i-th layer of the first model for:

[0067] Correspondingly, the output of each sample in the i-th layer of the first model is The output tensor data of the i-th layer is:

[0068] Perturbation data Ei-1,k As the output reasoning of layer i is passed to layer i+1, the output of layer i+1 is: Similarly, the perturbation data E i-1,k Pass it layer by layer to the I layer, and get the perturbation data E with the error level degradation coefficient k added to the i-th layer i-1,k The third final output tensor data after

[0069] Based on the first final output tensor data L I and the third final output tensor data The similarity between them is used to obtain the perturbation cumulative accuracy loss AP of the final layer corresponding to the error level degradation coefficient k of the i-th layer. I,i,k , Among them, 1≤i≤I, 0≤k≤K-1.

[0070] In step S1033, based on the layer disturbance accuracy loss and disturbance cumulative accuracy loss corresponding to each error level degradation coefficient of each layer, and combined with the layer reference accuracy loss, reference cumulative accuracy loss and layer data distribution statistics, the disturbance impact factor corresponding to each error level degradation coefficient of each layer is obtained. The disturbance impact factor corresponding to the error level degradation coefficient k of the i-th layer is IF i,k , whose expression is: Wherein, 1≤i≤I, 0≤k≤K-1, labs() is the absolute value function, lb() is the logarithm with base 2, and α and β are preset weighting coefficients. As an example, α=0.7 and β=0.3. Those skilled in the art may also select other values ​​of α and β according to actual circumstances, for example, α=0.65 and β=0.4.

[0071] In step S104, select K disturbance impact factors (IF i,k ,0≤k≤K-1) is the error level degradation coefficient k corresponding to the smallest disturbance influence factor, which is used as the best bit position of the layer, so as to obtain the best bit positions B1, B2, ..., B i ,…,B I For the best bit B of layer i i , and its calculation formula is: in, Indicates returning the index number (k value) corresponding to the smallest disturbance influence factor among the K disturbance influence factors of the i-th layer.

[0072] In step S105, based on the best bit positions B1, B2, ..., Bi ,…,B I The parameters of the corresponding layers of the first model are quantified respectively to obtain a target model that can be directly deployed on the end-side device.

[0073] It should be noted that the error level degradation coefficient can be a set of consecutive increasing integers as described in the above embodiment, or it can be a number of values ​​selected according to model application practices. As an example, k is selected only from even numbers such as 4, 6, 8, and 10. Without departing from the principles of this application, these modified or replaced technical solutions will fall within the scope of protection of this application.

[0074] As can be seen from the above embodiments, the optimal bit size for each layer can be equal to int8, or greater than or less than int8. For layers with low operator precision requirements, bit sizes lower than int8 can be used for quantization, thereby improving computing speed and storage efficiency, and improving the operating efficiency of the large end-model. For layers with high operator precision requirements, the quantization bit size can be appropriately increased (higher than int8) to compensate, to a certain extent, for the precision loss caused by int8 quantization, thereby ensuring the operating effect of the large end-model. In other words, the technical solution of this application achieves maximum utilization under the different layer precision requirements of the large end-model.

[0075] It should be noted that in the embodiments of this application, cosine similarity is used to evaluate the precision loss. In other embodiments, those skilled in the art may also use other similarity calculation methods such as sine similarity and Euclidean distance to evaluate the precision loss in this application. Without departing from the principles of this application, these modified or replaced technical solutions will fall within the scope of protection of this application.

[0076] Another aspect of the present application also provides a large model system on the end side, which is deployed on the end side device. Figure 5 As shown, the end-side large model system 5 includes a quantitative reasoning library 51, a quantitative optimization module 52 and an inference operation module 53.

[0077] The quantitative reasoning library 51 includes: operators for each layer in the first model, operators for each layer in the reference target model, and operators for each layer in the target model, and is responsible for the underlying operations of each model. The quantitative optimization module 52 is configured to obtain the target model based on the quantitative reasoning library 51 through the model lightweight method of the end-side large model described in any of the above embodiments. The reasoning operation module 53 is configured to control the operation and scheduling of the target model based on the quantitative reasoning library 51 to realize the functions of the end-side large model. As an example, the end-side device 5 is an edge computing terminal.

[0078] It should be noted that the target model can be obtained by a cloud server by executing the lightweight model quantization method described in any of the above embodiments for the large on-device model and then deployed on the device. Alternatively, the target model can be obtained by executing the lightweight model quantization method described in any of the above embodiments for the large on-device model system on the device through a quantization optimization module. Furthermore, when the target model is generated by the large on-device model system, it will have better compatibility.

[0079] It should be pointed out that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of the present application, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application, and therefore will also fall within the scope of protection of this application.

[0080] It will be understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment of the present application can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium can include: any entity or device, medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0081] Thus far, the technical solution of the present application has been described in conjunction with an embodiment shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of the present application is obviously not limited to these specific embodiments. Without departing from the principles of the present application, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present application.

Claims

1. A method for lightweighting a large end-side model, characterized in that: The method comprises: quantizing the first model based on the second bit to obtain a reference target model, wherein the first model is constructed based on the first bit and has been trained based on the first sample set; Performing inference on the first model based on the first inference sample set to obtain statistical results of layer data distribution of each layer; Inferring the first model and the reference target model based on the second inference sample set, and combining the K disturbance data corresponding to the K error level degradation coefficients of each layer and the statistical results of the layer data distribution to obtain K disturbance influence factors corresponding to the K error level degradation coefficients of each layer; Determining the optimal bit position of each layer based on the K disturbance impact factors of each layer; quantizing the first model based on the optimal bit positions of each layer to obtain a target model; The first reasoning sample set and the second reasoning sample set are respectively constructed based on the first sample set.

2. The method for lightweighting a large end-side model according to claim 1, characterized in that: The method for obtaining the disturbance impact factor includes: Performing inference on the first model and the reference target model based on the second inference sample set to obtain a layer reference accuracy loss of each layer and a reference cumulative accuracy loss of a final layer; Based on the second inference sample set and in combination with the K perturbation data corresponding to the K error level degradation coefficients of each layer, reasoning on the first model and the reference target model to obtain the layer perturbation accuracy loss corresponding to each error level degradation coefficient of each layer and the perturbation cumulative accuracy loss of the final layer; Based on the layer disturbance accuracy loss and the disturbance cumulative accuracy loss corresponding to each error level degradation coefficient of each layer, and combined with the layer reference accuracy loss, the reference cumulative accuracy loss and the layer data distribution statistical results, the disturbance impact factor corresponding to each error level degradation coefficient of each layer is obtained.

3. The method for lightweighting a large end-side model according to claim 2, characterized in that: The method for obtaining the layer reference accuracy loss includes: Obtain the output data l of the i-th layer of the reference target model corresponding to each sample in the second reasoning sample set i,n , get the first output tensor data L i ; Get the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set Will As the input data of the i-th layer of the reference target model, obtain the output data of the i-th layer of the reference target model corresponding to each sample in the second reasoning sample set Get the second output tensor data in, OOP i is the operator of the i-th layer of the reference target model; Based on the first output tensor data L i and the second output tensor data The similarity between them is used to obtain the reference accuracy loss of the i-th layer; Wherein, both the first model and the reference target model have I-layer structures, 1≤i≤I, The second inference sample set includes N samples, 0≤n≤N-1, and i and n are both integers.

4. The method for lightweighting a large end-side model according to claim 2, characterized in that: The first model and the reference target model both have a 1-layer structure, the final layer is the 1st layer, and the method for obtaining the reference cumulative precision loss of the final layer includes: Obtain the output data l of the first layer of the reference target model corresponding to each sample in the second reasoning sample set I,n , get the first final output tensor data L I ; Obtain the output data of the first layer of the first model corresponding to each sample in the second inference sample set Get the second final output tensor data Based on the first final output tensor data L I and the second final output tensor data The similarity between them is used to obtain the reference cumulative precision loss of the final layer; The second inference sample set includes N samples, 0≤n≤N-1, and n is an integer.

5. The method for lightweighting a large end-side model according to claim 2, characterized in that: The method for obtaining the layer perturbation accuracy loss includes: Obtain the output data l of the i-th layer of the reference target model corresponding to each sample in the second reasoning sample set i,n , get the first output tensor data L i ; Get the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set based on and the perturbation data E corresponding to the error level degradation coefficient k of the i-th layer i-1,k , construct the input data of the i-th layer of the reference target model Will As the input data of the i-th layer of the reference target model, obtain the output data l′ of the i-th layer of the reference target model corresponding to the error level degradation coefficient k i,n (E i-1,k ), get the third output tensor data L' i (E i-1,k ),in, OOP i is the operator of the i-th layer of the reference target model; Based on the first output tensor data L i and the third output tensor data L′ i (E i-1,k ), and obtain the layer disturbance accuracy loss corresponding to the error level degradation coefficient k of the i-th layer; Wherein, both the first model and the reference target model have an I-layer structure, 1≤i≤I; The second inference sample set includes N samples, 0≤n≤N-1; The number of the error level degradation coefficients is K, 0≤k≤K-1, and i, n and k are all integers.

6. The method for lightweighting a large end-side model according to claim 2, characterized in that: The first model and the reference target model both have a 1-layer structure, the final layer is the 1st layer, and the method for obtaining the disturbance cumulative precision loss of the final layer includes: Obtain the output data l of the first layer of the reference target model corresponding to each sample in the second reasoning sample set I,n , get the first final output tensor data L I ; Get the output data of the i-1th layer of the first model corresponding to each sample in the second inference sample set based on and the perturbation data E corresponding to the error level degradation coefficient k of the i-th layer i-1,k , construct the input data of the i-th layer of the first model Will As the input data of the i-th layer of the first model, obtain the output data of the i-th layer of the first model Get the third final output tensor data Based on the first final output tensor data L I and the third final output tensor data The similarity between them is used to obtain the disturbance cumulative accuracy loss of the final layer corresponding to the error level degradation coefficient k of the i-th layer; The second inference sample set includes N samples, 0≤n≤N-1; The number of the error level degradation coefficients is K, 0≤k≤K-1; 1≤i≤I, i, n and k are all integers.

7. The method for lightweighting a large end-side model according to claim 1, characterized in that: The method further comprises: The disturbance data is constructed based on a preset error initial value and a negative integer exponential power, wherein the negative integer exponential power is constructed based on the error level degradation coefficient k, wherein 0≤k≤K-1, and k is an integer.

8. The method for lightweighting a large end-side model according to any one of claims 1 to 7, characterized in that: The first sample set includes a first training sample set, a first test sample set, and a first validation sample set, and the method further includes: Constructing the first inference sample set based on the first training sample set, the first test sample set, extreme samples, and samples generated by an AI tool; Constructing the second reasoning sample set based on the first verification sample set; The samples in the first reasoning sample set and the samples in the second reasoning sample set are different, and the number of samples in the first reasoning sample set is greater than the number of samples in the second reasoning sample set.

9. The method for lightweighting a large end-side model according to claim 1, characterized in that: “Determining the optimal bit position of each layer based on the K disturbance impact factors of each layer” includes: Comparing the K disturbance impact factors of each layer respectively; The value of the error level degradation coefficient k corresponding to the disturbance influence factor with the smallest value is the optimal bit position of the layer, where 0≤k≤K-1, and k is an integer.

10. A large model system on the end side, characterized in that: The system comprises: A quantitative reasoning library, comprising: operators of each layer in the first model, operators of each layer in the reference target model, and operators of each layer in the target model; a quantization optimization module, the quantization optimization module being configured to obtain the target model based on the quantization inference library by using the model lightweight quantization method of the end-side large model according to any one of claims 1 to 9; An inference execution module is configured to control the execution and scheduling of the target model based on the quantized inference library.

Images